{"id":21076455,"url":"https://github.com/z77z/toutiaosplider","last_synced_at":"2026-02-08T23:34:59.090Z","repository":{"id":260016929,"uuid":"77418375","full_name":"z77z/TouTiaoSplider","owner":"z77z","description":"java spring+mybatis整合实现爬虫之《今日头条》搞笑动态图片爬取（详细）","archived":false,"fork":false,"pushed_at":"2016-12-27T02:09:38.000Z","size":16,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-20T22:53:13.461Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/z77z.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-12-27T02:07:22.000Z","updated_at":"2019-02-25T01:10:13.000Z","dependencies_parsed_at":"2024-10-29T04:38:34.831Z","dependency_job_id":null,"html_url":"https://github.com/z77z/TouTiaoSplider","commit_stats":null,"previous_names":["z77z/toutiaosplider"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/z77z%2FTouTiaoSplider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/z77z%2FTouTiaoSplider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/z77z%2FTouTiaoSplider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/z77z%2FTouTiaoSplider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/z77z","download_url":"https://codeload.github.com/z77z/TouTiaoSplider/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243521253,"owners_count":20304186,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T19:28:27.850Z","updated_at":"2026-02-08T23:34:51.053Z","avatar_url":"https://github.com/z77z.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"java spring+mybatis整合实现爬虫之《今日头条》搞笑动态图片爬取（详细）\n--------------------------------------\n\n\n----------\n\n一.此爬虫介绍\n-----\n\u003e今日头条本身就是做爬虫的，爬取各大网站的图片文字信息，再自己整合后推送给用户，特别是里面的动态图片，很有意思。在网上搜了搜，大多都是用Python来写的，本人是学习javaweb这块的，对正则表达式也不是很熟悉，就想着能不能换个我熟悉的方式来写。此爬虫使用spring+mybatis框架整合实现，使用mysql数据库保存爬取的数据，用jsoup来操作HTML的标签节点（完美避开正则表达式），获取文章中动态图片的链接，通过响应头中“Content-Type”的值来判断图片的格式，再将图片保存在本地。当然也可以爬取里面的文字，比如一些搞笑的黄段子，在此基础上稍加改动就可以实现，此爬虫只是提供一个入门的思路，更多好玩的爬虫玩法还待大家去开发，哈哈。\n\n二.技术选型\n------\n\n\u003e  1. 核心语言：java；\n\u003e  2. 核心框架：spring；\n\u003e  2. 持久层框架：mybatis；\n\u003e  3. 数据库连接池：Alibaba Drui；\n\u003e  4. 日志管理：Log4j；\n\u003e  5. jar包管理：maven； 。。。。\n\n三.找规律，划重点\n---------\n\n\u003e 打开头条首页，找到点击搞笑模块，点击F12,下滚后加载下一页，发现是通过ajax请求api来获取的数据，如下图：\n\n![这里写图片描述](http://img.blog.csdn.net/20161226223513868?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvcXFfMjA5NTQ5NTk=/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast)\n\n\u003e **这是响应的json数据，里面的参数和值顾名思义大家都懂得。**\n\n![这里写图片描述](http://img.blog.csdn.net/20161226223904416?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvcXFfMjA5NTQ5NTk=/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast)\n\n\u003e 是ajax访问就好解决了，通过我百度谷歌各种研究后发现，ajax请求的前三个参数是不变的，改变category参数是请求不同的模块，本列子是请求的搞笑模块所以值为funny，max_behot_time和max_behot_time_tmp这两个参数值是时间戳，首次请求是0，之后的值是响应json数据里面的next中的值。as和cp值是通过一段js生成的，其实就是一个加密了的时间戳而已。js代码后面会贴。\n\n四.开始搭框架撸代码\n-------\n\n\u003e 项目搭建后之后为下图所示的文件结构，不懂得自行谷歌  哈哈\n\n\n![这里写图片描述](http://img.blog.csdn.net/20161226225657064?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvcXFfMjA5NTQ5NTk=/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast)\n\n\u003e **不多说直接上核心代码了：**\n\n```\npackage io.z77z.main;\n\nimport io.z77z.dao.FunnyMapper;\nimport io.z77z.entity.Funny;\n\nimport java.io.BufferedInputStream;\nimport java.io.BufferedReader;\nimport java.io.FileOutputStream;\nimport java.io.FileReader;\nimport java.io.IOException;\nimport java.io.InputStreamReader;\nimport java.net.HttpURLConnection;\nimport java.net.URL;\nimport java.util.Date;\nimport java.util.UUID;\n\nimport javax.script.Invocable;\nimport javax.script.ScriptEngine;\nimport javax.script.ScriptEngineManager;\n\nimport org.jsoup.Connection;\nimport org.jsoup.Jsoup;\nimport org.jsoup.nodes.Document;\nimport org.jsoup.nodes.Element;\nimport org.jsoup.select.Elements;\nimport org.springframework.context.ApplicationContext;\nimport org.springframework.context.support.ClassPathXmlApplicationContext;\n\nimport com.alibaba.fastjson.JSON;\nimport com.alibaba.fastjson.JSONArray;\nimport com.alibaba.fastjson.JSONObject;\n\npublic class TouTiaoCrawler {\n\n\t// 搞笑板块的api地址\n\tpublic static final String FUNNY = \"http://www.toutiao.com/api/pc/feed/?utm_source=toutiao\u0026widen=1\";\n\n\t// 头条首页地址\n\tpublic static final String TOUTIAO = \"http://www.toutiao.com\";\n\n\t// 使用\"spring.xml\"和\"spring-mybatis.xml\"这两个配置文件创建Spring上下文\n\tstatic ApplicationContext ac = new ClassPathXmlApplicationContext(\n\t\t\t\"spring-mybatis.xml\");\n\n\t// 从Spring容器中根据bean的id取出我们要使用的funnyMapper对象\n\tstatic FunnyMapper funnyMapper = (FunnyMapper) ac.getBean(\"funnyMapper\");\n\n\t// 接口访问次数\n\tprivate static int refreshCount = 0;\n\n\t// 时间戳\n\tprivate static long time = 0;\n\n\tpublic static void main(String[] args) {\n\t\tSystem.out.println(\"----------开始干活！-----------------\");\n\t\twhile (true) {\n\t\t\tcrawler(time);\n\t\t}\n\t}\n\n\tpublic static void crawler(long hottime) {// 传入时间戳，会获取这个时间戳的内容\n\t\trefreshCount++;\n\t\tSystem.out.println(\"----------第\" + refreshCount + \"次刷新------返回的请求时间为：\"\n\t\t\t\t+ hottime + \"----------\");\n\t\tString url = FUNNY + \"\u0026max_behot_time=\" + hottime\n\t\t\t\t+ \"\u0026max_behot_time_tmp=\" + hottime;\n\t\tJSONObject param = getUrlParam(); // 获取用js代码得到的as和cp的值\n\t\t// 定义接口访问的模块\n\t\t/*\n\t\t * __all__ : 推荐 news_hot: 热点 funny：搞笑\n\t\t */\n\t\tString module = \"funny\";\n\t\turl += \"\u0026as=\" + param.get(\"as\") + \"\u0026cp=\" + param.get(\"cp\")\n\t\t\t\t+ \"\u0026category=\" + module;\n\t\tJSONObject json = null;\n\t\ttry {\n\t\t\tjson = getReturnJson(url);// 获取json串\n\t\t} catch (Exception e) {\n\t\t\te.printStackTrace();\n\t\t}\n\t\tif (json != null) {\n\t\t\ttime = json.getJSONObject(\"next\").getLongValue(\"max_behot_time\");\n\t\t\tJSONArray data = json.getJSONArray(\"data\");\n\t\t\tfor (int i = 0; i \u003c data.size(); i++) {\n\t\t\t\ttry {\n\t\t\t\t\tJSONObject obj = (JSONObject) data.get(i);\n\t\t\t\t\t// 判断这条文章是否已经爬过\n\t\t\t\t\tif (funnyMapper.selectByGroupId((String) obj\n\t\t\t\t\t\t\t.get(\"group_id\")) != null) {\n\t\t\t\t\t\tSystem.out\n\t\t\t\t\t\t\t\t.println(\"----------此文章已经爬过啦！-----------------\");\n\t\t\t\t\t\tcontinue;\n\t\t\t\t\t}\n\t\t\t\t\t// 访问页面返回document对象\n\t\t\t\t\tString url1 = TOUTIAO + \"/a\" + obj.getString(\"group_id\");\n\t\t\t\t\tDocument document = getArticleInfo(url1);\n\t\t\t\t\tSystem.out.println(\"----------成功访问了文章：\" + url1\n\t\t\t\t\t\t\t+ \"-----------------\");\n\t\t\t\t\t// 将document也存入\n\t\t\t\t\tobj.put(\"document\", document.toString());\n\t\t\t\t\t// 将json对象转换成java Entity对象\n\t\t\t\t\tFunny funny = JSON.parseObject(obj.toString(), Funny.class);\n\t\t\t\t\t// json入库\n\t\t\t\t\tfunny.setBehotTime(new Date());\n\t\t\t\t\tfunnyMapper.insertSelective(funny);\n\t\t\t\t} catch (Exception e) {\n\t\t\t\t\te.printStackTrace();\n\t\t\t\t}\n\t\t\t}\n\t\t} else {\n\t\t\tSystem.out.println(\"----------返回的json列表为空----------\");\n\t\t}\n\t}\n\n\t// 访问接口，返回json封装的数据格式\n\tpublic static JSONObject getReturnJson(String url) {\n\t\ttry {\n\t\t\tURL httpUrl = new URL(url);\n\t\t\tBufferedReader in = new BufferedReader(new InputStreamReader(\n\t\t\t\t\thttpUrl.openStream(), \"UTF-8\"));\n\t\t\tString line = null;\n\t\t\tString content = \"\";\n\t\t\twhile ((line = in.readLine()) != null) {\n\t\t\t\tcontent += line;\n\t\t\t}\n\t\t\tin.close();\n\t\t\treturn JSONObject.parseObject(content);\n\t\t} catch (Exception e) {\n\t\t\tSystem.err.println(\"访问失败:\" + url);\n\t\t\te.printStackTrace();\n\t\t}\n\t\treturn null;\n\t}\n\n\t// 获取网站的document对象\n\tpublic static Document getArticleInfo(String url) {\n\t\ttry {\n\t\t\tConnection connect = Jsoup.connect(url);\n\t\t\tDocument document;\n\t\t\tdocument = connect.get();\n\t\t\tElements article = document.getElementsByClass(\"article-content\");\n\t\t\tif (article.size() \u003e 0) {\n\t\t\t\tElements a = article.get(0).getElementsByTag(\"img\");\n\t\t\t\tif (a.size() \u003e 0) {\n\t\t\t\t\tfor (Element e : a) {\n\t\t\t\t\t\tString url2 = e.attr(\"src\");\n\t\t\t\t\t\t// 下载img标签里面的图片到本地\n\t\t\t\t\t\tsaveToFile(url2);\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\treturn document;\n\t\t} catch (IOException e) {\n\t\t\tSystem.err.println(\"访问文章页失败:\" + url + \"  原因\" + e.getMessage());\n\t\t\treturn null;\n\t\t}\n\t}\n\n\t// 执行js获取as和cp参数值\n\tpublic static JSONObject getUrlParam() {\n\t\tJSONObject jsonObject = null;\n\t\tFileReader reader = null;\n\t\ttry {\n\t\t\tScriptEngineManager manager = new ScriptEngineManager();\n\t\t\tScriptEngine engine = manager.getEngineByName(\"javascript\");\n\n\t\t\tString jsFileName = \"toutiao.js\"; // 读取js文件\n\n\t\t\treader = new FileReader(jsFileName); // 执行指定脚本\n\t\t\tengine.eval(reader);\n\n\t\t\tif (engine instanceof Invocable) {\n\t\t\t\tInvocable invoke = (Invocable) engine;\n\t\t\t\tObject obj = invoke.invokeFunction(\"getParam\");\n\t\t\t\tjsonObject = JSONObject.parseObject(obj != null ? obj\n\t\t\t\t\t\t.toString() : null);\n\t\t\t}\n\t\t} catch (Exception e) {\n\t\t\te.printStackTrace();\n\t\t} finally {\n\t\t\ttry {\n\t\t\t\tif (reader != null) {\n\t\t\t\t\treader.close();\n\t\t\t\t}\n\t\t\t} catch (IOException e) {\n\t\t\t\te.printStackTrace();\n\t\t\t}\n\t\t}\n\t\treturn jsonObject;\n\t}\n\n\t// 通过url获取图片并保存在本地\n\tpublic static void saveToFile(String destUrl) {\n\t\tFileOutputStream fos = null;\n\t\tBufferedInputStream bis = null;\n\t\tHttpURLConnection httpUrl = null;\n\t\tURL url = null;\n\t\tString uuid = UUID.randomUUID().toString();\n\t\tString fileAddress = \"d:\\\\imag/\" + uuid;// 存储本地文件地址\n\t\tint BUFFER_SIZE = 1024;\n\t\tbyte[] buf = new byte[BUFFER_SIZE];\n\t\tint size = 0;\n\t\ttry {\n\t\t\turl = new URL(destUrl);\n\t\t\thttpUrl = (HttpURLConnection) url.openConnection();\n\t\t\thttpUrl.connect();\n\t\t\tString Type = httpUrl.getHeaderField(\"Content-Type\");\n\t\t\tif (Type.equals(\"image/gif\")) {\n\t\t\t\tfileAddress += \".gif\";\n\t\t\t} else if (Type.equals(\"image/png\")) {\n\t\t\t\tfileAddress += \".png\";\n\t\t\t} else if (Type.equals(\"image/jpeg\")) {\n\t\t\t\tfileAddress += \".jpg\";\n\t\t\t} else {\n\t\t\t\tSystem.err.println(\"未知图片格式\");\n\t\t\t\treturn;\n\t\t\t}\n\t\t\tbis = new BufferedInputStream(httpUrl.getInputStream());\n\t\t\tfos = new FileOutputStream(fileAddress);\n\t\t\twhile ((size = bis.read(buf)) != -1) {\n\t\t\t\tfos.write(buf, 0, size);\n\t\t\t}\n\t\t\tfos.flush();\n\t\t\tSystem.out.println(\"图片保存成功！地址：\" + fileAddress);\n\t\t} catch (IOException e) {\n\t\t\te.printStackTrace();\n\t\t} catch (ClassCastException e) {\n\t\t\te.printStackTrace();\n\t\t} finally {\n\t\t\ttry {\n\t\t\t\tfos.close();\n\t\t\t\tbis.close();\n\t\t\t\thttpUrl.disconnect();\n\t\t\t} catch (IOException e) {\n\t\t\t\te.printStackTrace();\n\t\t\t} catch (NullPointerException e) {\n\t\t\t\te.printStackTrace();\n\t\t\t}\n\t\t}\n\t}\n}\n\n```\n\n\u003e **获取as和cp参数的js代码**\n\n```\nfunction getParam(){\n    var asas;\n    var cpcp;\n    var t = Math.floor((new Date).getTime() / 1e3)\n      , e = t.toString(16).toUpperCase()\n      , i = md5(t).toString().toUpperCase();\n    if (8 != e.length){\n        asas = \"479BB4B7254C150\";\n        cpcp = \"7E0AC8874BB0985\";\n    }else{\n        for (var n = i.slice(0, 5), o = i.slice(-5), a = \"\", s = 0; 5 \u003e s; s++){\n            a += n[s] + e[s];\n        }\n        for (var r = \"\", c = 0; 5 \u003e c; c++){\n            r += e[c + 3] + o[c];\n        }\n        asas = \"A1\" + a + e.slice(-3);\n        cpcp= e.slice(0, 3) + r + \"E1\";\n    }\n    return '{\"as\":\"'+asas+'\",\"cp\":\"'+cpcp+'\"}';\n}\n\n!function(e) {\n    \"use strict\";\n    function t(e, t) {\n        var n = (65535 \u0026 e) + (65535 \u0026 t)\n          , r = (e \u003e\u003e 16) + (t \u003e\u003e 16) + (n \u003e\u003e 16);\n        return r \u003c\u003c 16 | 65535 \u0026 n\n    }\n    function n(e, t) {\n        return e \u003c\u003c t | e \u003e\u003e\u003e 32 - t\n    }\n    function r(e, r, o, i, a, u) {\n        return t(n(t(t(r, e), t(i, u)), a), o)\n    }\n    function o(e, t, n, o, i, a, u) {\n        return r(t \u0026 n | ~t \u0026 o, e, t, i, a, u)\n    }\n    function i(e, t, n, o, i, a, u) {\n        return r(t \u0026 o | n \u0026 ~o, e, t, i, a, u)\n    }\n    function a(e, t, n, o, i, a, u) {\n        return r(t ^ n ^ o, e, t, i, a, u)\n    }\n    function u(e, t, n, o, i, a, u) {\n        return r(n ^ (t | ~o), e, t, i, a, u)\n    }\n    function s(e, n) {\n        e[n \u003e\u003e 5] |= 128 \u003c\u003c n % 32,\n        e[(n + 64 \u003e\u003e\u003e 9 \u003c\u003c 4) + 14] = n;\n        var r, s, c, l, f, p = 1732584193, d = -271733879, h = -1732584194, m = 271733878;\n        for (r = 0; r \u003c e.length; r += 16)\n            s = p,\n            c = d,\n            l = h,\n            f = m,\n            p = o(p, d, h, m, e[r], 7, -680876936),\n            m = o(m, p, d, h, e[r + 1], 12, -389564586),\n            h = o(h, m, p, d, e[r + 2], 17, 606105819),\n            d = o(d, h, m, p, e[r + 3], 22, -1044525330),\n            p = o(p, d, h, m, e[r + 4], 7, -176418897),\n            m = o(m, p, d, h, e[r + 5], 12, 1200080426),\n            h = o(h, m, p, d, e[r + 6], 17, -1473231341),\n            d = o(d, h, m, p, e[r + 7], 22, -45705983),\n            p = o(p, d, h, m, e[r + 8], 7, 1770035416),\n            m = o(m, p, d, h, e[r + 9], 12, -1958414417),\n            h = o(h, m, p, d, e[r + 10], 17, -42063),\n            d = o(d, h, m, p, e[r + 11], 22, -1990404162),\n            p = o(p, d, h, m, e[r + 12], 7, 1804603682),\n            m = o(m, p, d, h, e[r + 13], 12, -40341101),\n            h = o(h, m, p, d, e[r + 14], 17, -1502002290),\n            d = o(d, h, m, p, e[r + 15], 22, 1236535329),\n            p = i(p, d, h, m, e[r + 1], 5, -165796510),\n            m = i(m, p, d, h, e[r + 6], 9, -1069501632),\n            h = i(h, m, p, d, e[r + 11], 14, 643717713),\n            d = i(d, h, m, p, e[r], 20, -373897302),\n            p = i(p, d, h, m, e[r + 5], 5, -701558691),\n            m = i(m, p, d, h, e[r + 10], 9, 38016083),\n            h = i(h, m, p, d, e[r + 15], 14, -660478335),\n            d = i(d, h, m, p, e[r + 4], 20, -405537848),\n            p = i(p, d, h, m, e[r + 9], 5, 568446438),\n            m = i(m, p, d, h, e[r + 14], 9, -1019803690),\n            h = i(h, m, p, d, e[r + 3], 14, -187363961),\n            d = i(d, h, m, p, e[r + 8], 20, 1163531501),\n            p = i(p, d, h, m, e[r + 13], 5, -1444681467),\n            m = i(m, p, d, h, e[r + 2], 9, -51403784),\n            h = i(h, m, p, d, e[r + 7], 14, 1735328473),\n            d = i(d, h, m, p, e[r + 12], 20, -1926607734),\n            p = a(p, d, h, m, e[r + 5], 4, -378558),\n            m = a(m, p, d, h, e[r + 8], 11, -2022574463),\n            h = a(h, m, p, d, e[r + 11], 16, 1839030562),\n            d = a(d, h, m, p, e[r + 14], 23, -35309556),\n            p = a(p, d, h, m, e[r + 1], 4, -1530992060),\n            m = a(m, p, d, h, e[r + 4], 11, 1272893353),\n            h = a(h, m, p, d, e[r + 7], 16, -155497632),\n            d = a(d, h, m, p, e[r + 10], 23, -1094730640),\n            p = a(p, d, h, m, e[r + 13], 4, 681279174),\n            m = a(m, p, d, h, e[r], 11, -358537222),\n            h = a(h, m, p, d, e[r + 3], 16, -722521979),\n            d = a(d, h, m, p, e[r + 6], 23, 76029189),\n            p = a(p, d, h, m, e[r + 9], 4, -640364487),\n            m = a(m, p, d, h, e[r + 12], 11, -421815835),\n            h = a(h, m, p, d, e[r + 15], 16, 530742520),\n            d = a(d, h, m, p, e[r + 2], 23, -995338651),\n            p = u(p, d, h, m, e[r], 6, -198630844),\n            m = u(m, p, d, h, e[r + 7], 10, 1126891415),\n            h = u(h, m, p, d, e[r + 14], 15, -1416354905),\n            d = u(d, h, m, p, e[r + 5], 21, -57434055),\n            p = u(p, d, h, m, e[r + 12], 6, 1700485571),\n            m = u(m, p, d, h, e[r + 3], 10, -1894986606),\n            h = u(h, m, p, d, e[r + 10], 15, -1051523),\n            d = u(d, h, m, p, e[r + 1], 21, -2054922799),\n            p = u(p, d, h, m, e[r + 8], 6, 1873313359),\n            m = u(m, p, d, h, e[r + 15], 10, -30611744),\n            h = u(h, m, p, d, e[r + 6], 15, -1560198380),\n            d = u(d, h, m, p, e[r + 13], 21, 1309151649),\n            p = u(p, d, h, m, e[r + 4], 6, -145523070),\n            m = u(m, p, d, h, e[r + 11], 10, -1120210379),\n            h = u(h, m, p, d, e[r + 2], 15, 718787259),\n            d = u(d, h, m, p, e[r + 9], 21, -343485551),\n            p = t(p, s),\n            d = t(d, c),\n            h = t(h, l),\n            m = t(m, f);\n        return [p, d, h, m]\n    }\n    function c(e) {\n        var t, n = \"\";\n        for (t = 0; t \u003c 32 * e.length; t += 8)\n            n += String.fromCharCode(e[t \u003e\u003e 5] \u003e\u003e\u003e t % 32 \u0026 255);\n        return n\n    }\n    function l(e) {\n        var t, n = [];\n        for (n[(e.length \u003e\u003e 2) - 1] = void 0,\n        t = 0; t \u003c n.length; t += 1)\n            n[t] = 0;\n        for (t = 0; t \u003c 8 * e.length; t += 8)\n            n[t \u003e\u003e 5] |= (255 \u0026 e.charCodeAt(t / 8)) \u003c\u003c t % 32;\n        return n\n    }\n    function f(e) {\n        return c(s(l(e), 8 * e.length))\n    }\n    function p(e, t) {\n        var n, r, o = l(e), i = [], a = [];\n        for (i[15] = a[15] = void 0,\n        o.length \u003e 16 \u0026\u0026 (o = s(o, 8 * e.length)),\n        n = 0; 16 \u003e n; n += 1)\n            i[n] = 909522486 ^ o[n],\n            a[n] = 1549556828 ^ o[n];\n        return r = s(i.concat(l(t)), 512 + 8 * t.length),\n        c(s(a.concat(r), 640))\n    }\n    function d(e) {\n        var t, n, r = \"0123456789abcdef\", o = \"\";\n        for (n = 0; n \u003c e.length; n += 1)\n            t = e.charCodeAt(n),\n            o += r.charAt(t \u003e\u003e\u003e 4 \u0026 15) + r.charAt(15 \u0026 t);\n        return o\n    }\n    function h(e) {\n        return unescape(encodeURIComponent(e))\n    }\n    function m(e) {\n        return f(h(e))\n    }\n    function g(e) {\n        return d(m(e))\n    }\n    function v(e, t) {\n        return p(h(e), h(t))\n    }\n    function y(e, t) {\n        return d(v(e, t))\n    }\n    function b(e, t, n) {\n        return t ? n ? v(t, e) : y(t, e) : n ? m(e) : g(e)\n    }\n    \"function\" == typeof define \u0026\u0026 define.amd ? define(\"static/js/lib/md5\", [\"require\"], function() {\n        return b\n    }) : \"object\" == typeof module \u0026\u0026 module.exports ? module.exports = b : e.md5 = b\n}(this)\n```\n\n五.最后\n------\n\n\u003e 我还发现了头条有个简约版，研究后发现这个简约版应该更好爬一些。\n\n![这里写图片描述](http://img.blog.csdn.net/20161226230850348?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvcXFfMjA5NTQ5NTk=/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast)\n\n\u003e 访问的格式是p+页码，直接读取每页里面的链接，就可以进行爬取了，就不再通过json串来获取文章地址，也不需要传什么限制参数，在本项目上稍加改动就可以了\n\n![这里写图片描述](http://img.blog.csdn.net/20161226230958084?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvcXFfMjA5NTQ5NTk=/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast)\n\n![这里写图片描述](http://img.blog.csdn.net/20161226231623684?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvcXFfMjA5NTQ5NTk=/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast)\n\n\n六.JUST DO IT\n------------\n\n\u003e 。。。。。。。。。。。。。。。。。。。。。。\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fz77z%2Ftoutiaosplider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fz77z%2Ftoutiaosplider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fz77z%2Ftoutiaosplider/lists"}