Skip to content
This repository has been archived by the owner on Apr 20, 2021. It is now read-only.

Commit

Permalink
Merge branch 'dev'
Browse files Browse the repository at this point in the history
  • Loading branch information
ZhaoQi99 committed Dec 24, 2018
2 parents c99a4aa + c788238 commit 93fdd9e
Show file tree
Hide file tree
Showing 718 changed files with 43,651 additions and 124,246 deletions.
9 changes: 2 additions & 7 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,8 @@ sudo: true
python:
- "3.6"
install:
- pip install twilio
- pip install pypyodbc
- pip install -r requirements.txt
script:
- cp src/* test/
- cd test
- python test_spider.py
- python test_spider.py
- python test_tool.py
- python test_tool.py
- python test_single_spider.py
- /usr/bin/time -v python test_spider.py
21 changes: 16 additions & 5 deletions Pydoc/configs.html
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,13 @@
<font color="#ffffff" face="helvetica, arial">&nbsp;<br><big><big><strong>configs</strong></big></big></font></td
><td align=right valign=bottom
><font color="#ffffff" face="helvetica, arial"><a href=".">index</a><br><a href="file:f%3A%5Ccode%5Cpython%5Cschool_notice%5Csrc%5Cconfigs.py">f:\code\python\school_notice\src\configs.py</a></font></td></tr></table>
<p><tt>Created&nbsp;on&nbsp;Sep&nbsp;19,&nbsp;2018<br>
&nbsp;<br>
@author:&nbsp;QiZhao<br>
<p><tt>@author:&nbsp;QiZhao<br>
@contact:&nbsp;[email protected]<br>
@since:&nbsp;2018-09-19<br>
@license:&nbsp;GNU&nbsp;GPLv3<br>
@version:&nbsp;0.2.0</tt></p>
@version:&nbsp;0.3.0<br>
@LastModifiedBy:&nbsp;QiZhao<br>
@LastModifiedDate:&nbsp;2018-12-24</tt></p>
<p>
<table width="100%" cellspacing=0 cellpadding=2 border=0 summary="section">
<tr bgcolor="#55aa55">
Expand All @@ -22,16 +24,25 @@

<tr><td bgcolor="#55aa55"><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</tt></td><td>&nbsp;</td>
<td width="100%"><strong>ACCOUNT_ID</strong> = ''<br>
<strong>APPID</strong> = ' '<br>
<strong>AUTHOR_EMAIL</strong> = ''<br>
<strong>AUTHOR_NAME</strong> = ''<br>
<strong>AUTH_TOKEN</strong> = ''<br>
<strong>DATABASE_NAME</strong> = ''<br>
<strong>EMAIL_PORT</strong> = 0<br>
<strong>EMAIL_SERVER</strong> = ''<br>
<strong>FROM_ADDR</strong> = ''<br>
<strong>GRANT_TYPE</strong> = 'client_credential'<br>
<strong>LOG_ENABLED</strong> = True<br>
<strong>PASSWORD</strong> = ''<br>
<strong>SAVE_TYPE</strong> = 'MYSQL'<br>
<strong>SCHOOL_NAME</strong> = ''<br>
<strong>SPIDER_CONFIG</strong> = [{'coding': '', 'rule': '', 'subject_CN': '', 'subject_EN': '', 'url': '', 'url_main': ''}, {'coding': '', 'rule': '', 'subject_CN': '', 'subject_EN': '', 'url': '', 'url_main': ''}]<br>
<strong>SECRET</strong> = ' '<br>
<strong>SHOW_RIGHT</strong> = False<br>
<strong>SPIDER_CONFIG</strong> = []<br>
<strong>SQL_PASSWORD</strong> = ''<br>
<strong>SQL_USERNAME</strong> = ''<br>
<strong>TARGET_IP</strong> = ''<br>
<strong>TWILIO_NUMBER</strong> = ''<br>
<strong>VERSION</strong> = ''</td></tr></table>
</body></html>
8 changes: 7 additions & 1 deletion Pydoc/main.html
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,13 @@
<font color="#ffffff" face="helvetica, arial">&nbsp;<br><big><big><strong>main</strong></big></big></font></td
><td align=right valign=bottom
><font color="#ffffff" face="helvetica, arial"><a href=".">index</a><br><a href="file:f%3A%5Ccode%5Cpython%5Cschool_notice%5Csrc%5Cmain.py">f:\code\python\school_notice\src\main.py</a></font></td></tr></table>
<p><tt>#&nbsp;encoding='utf-8'</tt></p>
<p><tt>@author:&nbsp;QiZhao<br>
@contact:&nbsp;[email protected]<br>
@since:&nbsp;2018-05-08<br>
@license:&nbsp;GNU&nbsp;GPLv3<br>
@version:&nbsp;0.3.0<br>
@LastModifiedBy:&nbsp;QiZhao<br>
@LastModifiedDate:&nbsp;2018-10-27</tt></p>
<p>
<table width="100%" cellspacing=0 cellpadding=2 border=0 summary="section">
<tr bgcolor="#aa55cc">
Expand Down
31 changes: 22 additions & 9 deletions Pydoc/send.html
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,13 @@
<font color="#ffffff" face="helvetica, arial">&nbsp;<br><big><big><strong>send</strong></big></big></font></td
><td align=right valign=bottom
><font color="#ffffff" face="helvetica, arial"><a href=".">index</a><br><a href="file:f%3A%5Ccode%5Cpython%5Cschool_notice%5Csrc%5Csend.py">f:\code\python\school_notice\src\send.py</a></font></td></tr></table>
<p></p>
<p><tt>@author:&nbsp;QiZhao<br>
@contact:&nbsp;[email protected]<br>
@since:&nbsp;2018-05-07<br>
@license:&nbsp;GNU&nbsp;GPLv3<br>
@version:&nbsp;0.3.0<br>
@LastModifiedBy:&nbsp;QiZhao<br>
@LastModifiedDate:&nbsp;2018-10-24</tt></p>
<p>
<table width="100%" cellspacing=0 cellpadding=2 border=0 summary="section">
<tr bgcolor="#aa55cc">
Expand All @@ -18,19 +24,22 @@

<tr><td bgcolor="#aa55cc"><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</tt></td><td>&nbsp;</td>
<td width="100%"><table width="100%" summary="list"><tr><td width="25%" valign=top><a href="configs.html">configs</a><br>
</td><td width="25%" valign=top><a href="smtplib.html">smtplib</a><br>
</td><td width="25%" valign=top><a href="twilio.html">twilio</a><br>
<a href="json.html">json</a><br>
</td><td width="25%" valign=top><a href="requests.html">requests</a><br>
<a href="smtplib.html">smtplib</a><br>
</td><td width="25%" valign=top><a href="time.html">time</a><br>
<a href="twilio.html">twilio</a><br>
</td><td width="25%" valign=top></td></tr></table></td></tr></table><p>
<table width="100%" cellspacing=0 cellpadding=2 border=0 summary="section">
<tr bgcolor="#eeaa77">
<td colspan=3 valign=bottom>&nbsp;<br>
<font color="#ffffff" face="helvetica, arial"><big><strong>Functions</strong></big></font></td></tr>

<tr><td bgcolor="#eeaa77"><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</tt></td><td>&nbsp;</td>
<td width="100%"><dl><dt><a name="-Send"><strong>Send</strong></a>(msgs, subject, send_number, to_addr_str, flag=True)</dt><dd><tt>向手机号码为send_number的人发送通知信息<br>
<td width="100%"><dl><dt><a name="-Send"><strong>Send</strong></a>(msgs, subject, send_number, to_addr_str, message_type, flag=True)</dt><dd><tt>向手机号码为send_number的人发送通知信息<br>
向to_addr_str中的邮箱地址发送主题为subject的通知信息<br>
支持是否写入日志记录的选择&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>
&nbsp;<br>
Args:<br>
&nbsp;&nbsp;&nbsp;&nbsp;msgs:&nbsp;存储要发送的内容的列表,且该列表的每个元素为字典,<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;列表元素中的字典必须包含三个键值对,且key必须为'title','link','date',<br>
Expand All @@ -40,6 +49,7 @@
&nbsp;&nbsp;&nbsp;&nbsp;send_number:&nbsp;短信接收者的手机号码<br>
&nbsp;&nbsp;&nbsp;&nbsp;to_addr_str:&nbsp;收件人的邮箱地址,多个邮箱地址之间应以','分割,类型为字符串<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;例如:'[email protected]','[email protected],[email protected]'<br>
&nbsp;&nbsp;&nbsp;&nbsp;message_type:&nbsp;类型(通知/新闻)<br>
&nbsp;&nbsp;&nbsp;&nbsp;flag:&nbsp;一个可选变量,用来决定是否在发送日志中记录此次发送信息,默认为True(记录)</tt></dd></dl>
<dl><dt><a name="-Send_email"><strong>Send_email</strong></a>(txt, to_addr_str, subject)</dt><dd><tt>向to_addr_str中的邮箱地址发送主题为subject,正文部分为txt的邮件<br>
支持多人同时发送<br>
Expand All @@ -48,25 +58,28 @@
&nbsp;&nbsp;&nbsp;&nbsp;to_add_str:&nbsp;收件人的邮箱地址,多个邮箱地址之间应以','分割,类型为字符串<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;例如:'[email protected]','[email protected],[email protected]'<br>
&nbsp;&nbsp;&nbsp;&nbsp;subject:&nbsp;邮件主题<br>
&nbsp;&nbsp;&nbsp;&nbsp;<br>
&nbsp;<br>
Returns:<br>
&nbsp;&nbsp;&nbsp;&nbsp;log_send_email:&nbsp;经过处理的含收件人邮箱地址以及发送结果的日志信息,类型为字符串<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;例如:'[email protected]&nbsp;邮件发送成功!'<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'[email protected]&nbsp;邮件发送失败,请检查你的账号是否有效或网络是否良好!'<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'[email protected],[email protected]&nbsp;邮件发送成功!'</tt></dd></dl>
<dl><dt><a name="-Send_sms"><strong>Send_sms</strong></a>(send_number, msg)</dt><dd><tt>向手机号码为send_number的人发送内容为msg的短信<br>
<dl><dt><a name="-Send_sms"><strong>Send_sms</strong></a>(send_number, msg)</dt><dd><tt>向手机号码为send_number的人发送内容为msg的短信&nbsp;<br>
支持多个手机号码的发送<br>
Args:<br>
&nbsp;&nbsp;&nbsp;&nbsp;send_number:&nbsp;短信接收者的手机号码<br>
&nbsp;&nbsp;&nbsp;&nbsp;msg:&nbsp;要发送的文本内容,类型为字符串<br>
&nbsp;&nbsp;&nbsp;&nbsp;<br>
&nbsp;<br>
Returns:<br>
&nbsp;&nbsp;&nbsp;&nbsp;经过处理的含短信接收者手机号码以及发送结果的日志信息,类型为字符串<br>
&nbsp;&nbsp;&nbsp;&nbsp;例如:<br>
&nbsp;&nbsp;&nbsp;&nbsp;'+8615012345678&nbsp;短信已经发送'<br>
&nbsp;&nbsp;&nbsp;&nbsp;'+8615012345678&nbsp;短信发送失败,请检查你的账号是否有效或网络是否良好!'<br>
&nbsp;&nbsp;&nbsp;&nbsp;<br>
&nbsp;<br>
send_number应已经在twilio上验证过<br>
msg中不能包含敏感词,否则短信会被运营商拦截</tt></dd></dl>
<dl><dt><a name="-get_token"><strong>get_token</strong></a>()</dt><dd><tt>#&nbsp;获取微信access_token<br>
#&nbsp;TODO&nbsp;每两小时token会过期,一天大概只能获取2000次</tt></dd></dl>
<dl><dt><a name="-send_to_wechat"><strong>send_to_wechat</strong></a>(str='default_words!')</dt><dd><tt>#&nbsp;发送消息给订阅号(订阅号由get_token决定</tt></dd></dl>
</td></tr></table>
</body></html>
54 changes: 29 additions & 25 deletions Pydoc/spider.html
Original file line number Diff line number Diff line change
Expand Up @@ -9,60 +9,64 @@
<font color="#ffffff" face="helvetica, arial">&nbsp;<br><big><big><strong>spider</strong></big></big></font></td
><td align=right valign=bottom
><font color="#ffffff" face="helvetica, arial"><a href=".">index</a><br><a href="file:f%3A%5Ccode%5Cpython%5Cschool_notice%5Csrc%5Cspider.py">f:\code\python\school_notice\src\spider.py</a></font></td></tr></table>
<p><tt>Created&nbsp;on&nbsp;Mar&nbsp;7,&nbsp;2018<br>
&nbsp;<br>
@author:&nbsp;QiZhao<br>
@license:&nbsp;GNU&nbsp;GPLv3<br>
@version:&nbsp;0.2.0</tt></p>
<p><tt>#&nbsp;encoding='utf-8'</tt></p>
<p>
<table width="100%" cellspacing=0 cellpadding=2 border=0 summary="section">
<tr bgcolor="#aa55cc">
<td colspan=3 valign=bottom>&nbsp;<br>
<font color="#ffffff" face="helvetica, arial"><big><strong>Modules</strong></big></font></td></tr>

<tr><td bgcolor="#aa55cc"><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</tt></td><td>&nbsp;</td>
<td width="100%"><table width="100%" summary="list"><tr><td width="25%" valign=top><a href="re.html">re</a><br>
<td width="100%"><table width="100%" summary="list"><tr><td width="25%" valign=top><a href="configs.html">configs</a><br>
<a href="re.html">re</a><br>
</td><td width="25%" valign=top><a href="sqlhelper.html">sqlhelper</a><br>
</td><td width="25%" valign=top><a href="tool.html">tool</a><br>
<a href="tool.html">tool</a><br>
</td><td width="25%" valign=top><a href="urllib.html">urllib</a><br>
</td></tr></table></td></tr></table><p>
</td><td width="25%" valign=top></td></tr></table></td></tr></table><p>
<table width="100%" cellspacing=0 cellpadding=2 border=0 summary="section">
<tr bgcolor="#eeaa77">
<td colspan=3 valign=bottom>&nbsp;<br>
<font color="#ffffff" face="helvetica, arial"><big><strong>Functions</strong></big></font></td></tr>

<tr><td bgcolor="#eeaa77"><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</tt></td><td>&nbsp;</td>
<td width="100%"><dl><dt><a name="-Data_processing"><strong>Data_processing</strong></a>(subject_EN, data, url_main)</dt><dd><tt>读取数据文件,并将新抓取的通知信息中的链接部分处理为长链接,<br>
然后以通知链接为参照,与数据库中的数据进行对比,并将新通知写入数据库,<br>
<td width="100%"><dl><dt><a name="-Data_processing"><strong>Data_processing</strong></a>(department_EN, data, url_main, message_type)</dt><dd><tt>读取数据文件,并将新抓取的通知信息中的链接部分处理为长链接,<br>
然后以通知链接为参照,与文件/数据库中的数据进行对比,并将新通知写入文件/数据库,<br>
返回检查更新的状态码与处理后的数据<br>
&nbsp;<br>
Args:<br>
&nbsp;&nbsp;&nbsp;&nbsp;subject_EN:&nbsp;生成的数据文件的文件名<br>
&nbsp;&nbsp;&nbsp;&nbsp;department_EN:&nbsp;生成的数据文件的文件名<br>
&nbsp;&nbsp;&nbsp;&nbsp;data:&nbsp;存储通知主要内容的列表,且该列表每个元素为字典<br>
&nbsp;&nbsp;&nbsp;&nbsp;例如:[{'title':'关于xxx的通知','date':'2017-03-10','link':'id=5'},<br>
&nbsp;&nbsp;&nbsp;&nbsp;{'title':'关于xxx的通知','date':'2017-03-10','link':'id=5'}]<br>
&nbsp;&nbsp;&nbsp;&nbsp;url_main:&nbsp;单条通知的url的公共部分<br>
&nbsp;&nbsp;&nbsp;&nbsp;message_type:&nbsp;类型(通知/新闻)<br>
Returns:<br>
&nbsp;&nbsp;&nbsp;&nbsp;status:&nbsp;检查更新的状态码,无新通知时为0,首次抓取为-1,有新通知时通知条数<br>
&nbsp;&nbsp;&nbsp;&nbsp;new_data:&nbsp;存储经处理后的通知内容的列表,且该列表每个元素为字典<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;例如:[{'title':'关于xxx的通知','date':'2017-03-10','link':'<a href="http://xxxx.com">http://xxxx.com</a>'},<br>
&nbsp;&nbsp;&nbsp;&nbsp;{'title':'关于xxx的通知','date':'2017-03-10','link':'<a href="http://xxxx.com">http://xxxx.com</a>‘}]</tt></dd></dl>
<dl><dt><a name="-Log_generate"><strong>Log_generate</strong></a>(status, data, subject_CN)</dt><dd><tt>&nbsp;&nbsp;&nbsp;&nbsp;依据检查更新的结果,生成不同的日志内容,并返回日志内容<br>
<dl><dt><a name="-Log_generate"><strong>Log_generate</strong></a>(status, data, department_CN, message_type)</dt><dd><tt>依据检查更新的结果,生成不同的日志内容,并返回日志内容<br>
&nbsp;<br>
&nbsp;&nbsp;&nbsp;&nbsp;Args:<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;data:存储通知提醒主要内容的列表,且该列表每个元素为字典<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;例如:[{'title':'关于xxx的通知','date':'2017-03-10','link':'<a href="http://xxxx.com">http://xxxx.com</a>'},<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{'title':'关于xxx的通知','date':'2017-03-10','link':'<a href="http://xxxx.com">http://xxxx.com</a>'}]<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;subject_CN:&nbsp;抓取的网站类型<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;status:&nbsp;检查更新的状态码<br>
Args:<br>
&nbsp;&nbsp;&nbsp;&nbsp;data:存储通知提醒主要内容的列表,且该列表每个元素为字典<br>
&nbsp;&nbsp;&nbsp;&nbsp;例如:[{'title':'关于xxx的通知','date':'2017-03-10','link':'<a href="http://xxxx.com">http://xxxx.com</a>'},<br>
&nbsp;&nbsp;&nbsp;&nbsp;{'title':'关于xxx的通知','date':'2017-03-10','link':'<a href="http://xxxx.com">http://xxxx.com</a>'}]<br>
&nbsp;&nbsp;&nbsp;&nbsp;department_CN:&nbsp;抓取的部门名称<br>
&nbsp;&nbsp;&nbsp;&nbsp;status:&nbsp;检查更新的状态码<br>
&nbsp;<br>
&nbsp;&nbsp;&nbsp;&nbsp;Returns:<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;log_txt:&nbsp;日志的主要内容,类型为字符串或每个元素均为列表的列表,且元素列表的元素均为字符串。<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;例如:'首次抓取师师大主页!<br>
','师大主页暂无新通知!<br>
'<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[['关于xxx的通知','2017-03-10','<a href="http://xxxx.com'],['关于xxx的通知','2017-03-10','http://xxxx.com">http://xxxx.com'],['关于xxx的通知','2017-03-10','http://xxxx.com</a>']]</tt></dd></dl>
<dl><dt><a name="-Spider"><strong>Spider</strong></a>(url, url_main, rule, subject_CN, subject_EN, coding, flag=True)</dt><dd><tt>爬取url的源码,并从中按照rule提供的正则表达式规则提取有用信息,并对数据进行处理,<br>
Returns:<br>
&nbsp;&nbsp;&nbsp;&nbsp;log_txt:&nbsp;日志的主要内容,类型为字符串或每个元素均为列表的列表,且元素列表的元素均为字符串。<br>
&nbsp;&nbsp;&nbsp;&nbsp;例如:'首次抓取师师大主页!','师大主页暂无新通知!'<br>
&nbsp;&nbsp;&nbsp;&nbsp;[['关于xxx的通知','2017-03-10','<a href="http://xxxx.com'],['关于xxx的通知','2017-03-10','http://xxxx.com">http://xxxx.com'],['关于xxx的通知','2017-03-10','http://xxxx.com</a>']]</tt></dd></dl>
<dl><dt><a name="-Save"><strong>Save</strong></a>(new_data, department_EN, message_type)</dt><dd><tt>将新抓取到的通知信息写入数据文件或数据库中<br>
&nbsp;<br>
Args:<br>
&nbsp;&nbsp;&nbsp;&nbsp;new_data:&nbsp;存储经处理后的通知内容的列表,且该列表每个元素为字典<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;例如:[{'title':'关于xxx的通知','date':'2017-03-10','link':'<a href="http://xxxx.com">http://xxxx.com</a>'},<br>
&nbsp;&nbsp;&nbsp;&nbsp;{'title':'关于xxx的通知','date':'2017-03-10','link':'<a href="http://xxxx.com">http://xxxx.com</a>‘}]<br>
&nbsp;&nbsp;&nbsp;&nbsp;department_EN:&nbsp;生成的数据文件的文件名<br>
&nbsp;&nbsp;&nbsp;&nbsp;message_type:&nbsp;类型(通知/新闻)</tt></dd></dl>
<dl><dt><a name="-Spider"><strong>Spider</strong></a>(dic, flag=True)</dt><dd><tt>爬取url的源码,并从中按照rule提供的正则表达式规则提取有用信息,并对数据进行处理,<br>
生成通知提醒的内容,在subject_EN+'_log.md'文件中记录日志,<br>
返回检查更新的状态码,以及通知提醒的内容<br>
若无新通知,则通知提醒的内容为空<br>
Expand Down
Loading

0 comments on commit 93fdd9e

Please sign in to comment.