Skip to content

Commit

Permalink
Merge pull request #29 from knownsec/dev
Browse files Browse the repository at this point in the history 
Dev
  • Loading branch information
@wh0am1i
wh0am1i authored Mar 23, 2021
2 parents e5a22b6 + 95dcaf2 commit 09fa02b
Show file tree
Hide file tree
Showing 10 changed files with 756 additions and 131 deletions.
5 changes: 4 additions & 1 deletion CHANGELOG
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Released]
## [2.0.4.2] - 2021-03-22
- add new CLI command history [-h] [-filter filed=regexp] [-force] [-num value] ip
- add new CLI command clear [-h] [-setting] [-cache]
- command search add new parameters [-force]

## [Unreleased]
## [2.0.4.0] - 2021-02-05
- add new CLI command: -[figure {pie,hist}]
- fix some bug
Expand Down
204 changes: 174 additions & 30 deletions README.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,16 +37,18 @@ After successfully installing ``ZoomEye-python``, you can use the
::

$ zoomeye -h
usage: cli.py [-h] {info,search,init} ...
usage: zoomeye [-h] {info,search,init,history,clear} ...

positional arguments:
{info,search,init}
info Show ZoomEye account info
search Search the ZoomEye database
init Initialize the token for ZoomEye-python
positional arguments:
{info,search,init,history,clear}
info Show ZoomEye account info
search Search the ZoomEye database
init Initialize the token for ZoomEye-python
history Query device history
clear Manually clear the cache and user information

optional arguments:
-h, --help show this help message and exit
optional arguments:
-h, --help show this help message and exit

1.initialize token
^^^^^^^^^^^^^^^^^^
Expand Down Expand Up @@ -132,6 +134,7 @@ will explain and demonstrate below.
-stat the distribution of statistical data result sets
-filter query the list of a certain area in the data result set, or filter according to the content
-save the result set can be exported according to the filter conditions
-force ignore the local cache and force the data to be obtained from the API

4.number of data
^^^^^^^^^^^^^^^^
Expand Down Expand Up @@ -201,6 +204,7 @@ devices:
Linux telnetd 3
Pocket CMD telnetd 1


6.data filter
^^^^^^^^^^^^^

Expand All @@ -218,6 +222,7 @@ by this command include:
country show country details
asn show as number details
banner show details of characteristic response
time show record data time
* when this symbol is included, show all field details

Compared to the omitted display by default, the complete data can be
Expand All @@ -244,6 +249,7 @@ supported), and the format is ``field=regexp``, for example, we query in

total: 1


7.data export
^^^^^^^^^^^^^

Expand All @@ -267,27 +273,145 @@ the format of line json, as follows:
the file can be as input, it is parsed and processed again through
``cli``, such as ``zoomeye search "xxxxx.json"``.


8.graphical data
^^^^^^^^^^^^^^^^

The ``-figure`` parameter is a data visualization parameter. This parameter provides two display methods: ``pie (pie chart)`` and ``hist (histogram)``. The data will still be displayed without specifying it. When ``-figure`` is specified , Only graphics will be displayed. The pie chart is as follows:

|image-20210205004653480|
|image-20210205005016399|
.. figure:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210205004653480.png
:width: 500px

.. figure:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210205005016399.png
:width: 500px

The histogram is as follows:

|image-20210205004806739|
|image-20210205005117712|
.. figure:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210205004806739.png
:width: 500px

9.data cache
^^^^^^^^^^^^
.. figure:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210205005117712.png
:width: 500px


9. IP history
^^^^^^^^^^^^^

``ZoomEye-python`` provides the function of querying IP historical device data. Use the command ``history [ip]`` to query the historical data of IP devices. The usage is as follows:

::

$zoomeye history "207.xx.xx.13" -num 1
207.xx.xx.13
Hostnames: [unknown]
Country: United States
City: Lake Charles
Organization: fulair.com
Lastupdated: 2021-02-18T03:44:06
Number of open ports: 1
Number of historical probes: 1

timestamp port/service app raw_data
2021-02-18 03:44:06 80/http Apache httpd HTTP/1.0 301 Moved Permanently...



By default, five fields are shown to users:

::

1. time recorded time
2. service open service
3. port port
4. app web application
5. raw fingerprint information


Use ``zoomeye history -h`` to view the parameters provided by ``history``.

::

$zoomeye history -h

usage: zoomeye history [-h] [-filter filed=regexp] [-force] ip

positional arguments:
ip search historical device IP

optional arguments:
-h, --help show this help message and exit
-filter filed=regexp filter data and print raw data detail. field:
[time,port,service,app,raw]
-force ignore the local cache and force the data to be
obtained from the API

The following is a demonstration of ``-filter``:


::

$zoomeye history "207.xx.xx.13" -filter "time=^2019-08,port,service"
207.xx.xx.13
Hostnames: [unknown]
Country: United States
City: Lake Charles
Organization: fulair.com
Lastupdated: 2019-08-16T10:53:46
Number of open ports: 3
Number of historical probes: 3

time port service
2019-08-16 10:53:46 389 ldap
2019-08-08 23:32:30 22 ssh
2019-08-03 01:55:59 80 http


The `-filter` parameter supports the filtering of the following five fields:

::

1.time scan time
2.port port information
3.service open service
4.app web application
5.raw original fingerprint information
* when this symbol is included, show all field details


A display of the ``id`` field is added during the display. ``id`` is the serial number. For the convenience of viewing, it cannot be used as a filtered field.

..
Note: At present, only the above five fields are allowed to filter.

The user quota will also be consumed when using the ``history`` command. The user quota will be deducted for the number of pieces of data returned in the ``history`` command. For example: IP "8.8.8.8" has a total of ``944`` historical records, and the user quota of ``944`` is deducted for one query.

10.cleanup function
^^^^^^^^^^^^^^^^^^^^

Users search for a large amount of data every day, which causes the storage space occupied by the cache folder to gradually increase; if users use ``ZoomEye-python`` on a public server, it may cause their own ``API KEY`` and ``ACCESS TOKEN`` to leak .
For this reason, ``ZoomEye-python`` provides the clear command ``zoomeye clear``, which can clear the cached data and user configuration. The usage is as follows:

::

$zoomeye clear -h
usage: zoomeye clear [-h] [-setting] [-cache]

optional arguments:
-h, --help show this help message and exit
-setting clear user api key and access token
-cache clear local cache file


11.data cache
^^^^^^^^^^^^^

``ZoomEye-python`` provides a caching in ``cli`` mode, which is located
under ``~/.config/zoomeye/cache`` to save user quota as much as
possible; the data set that the user has queried will be cached locally
for 5 days. when users query the same data set, quotas are not consumed.


0x03 video
~~~~~~~~~~

Expand Down Expand Up @@ -425,6 +549,7 @@ data more conveniently and extract the specified data fields as follows:
``host-search``: app / version / device / ip / port / hostname / city
/ country / asn / banner
0x05 contributions
~~~~~~~~~~~~~~~~~~
Expand All @@ -449,8 +574,13 @@ data more conveniently and extract the specified data fields as follows:
| **2.How to enter dork with quotes?**
| When using cli to search, you will encounter dork with quotes, for example: ``"<body style=\"margin:0;padding:0\"> <p align=\"center\"> <iframe src=\ "index.xhtml\""``, when dork contains quotation marks or multiple quotation marks, the outermost layer of dork must be wrapped in quotation marks to indicate a parameter as a whole, otherwise command line parameter parsing will cause problems. Then the correct search method for the following dork should be: ``'"<body style=\"margin:0;padding:0\"> <p align=\"center\"> <iframe src=\"index.xhtml\" "'``.
| |image-20210205131713799|
| |image-20210205131802799|
.. figure:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210205131713799.png
:width: 500px
.. figure:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210205131802799.png
:width: 500px
| **3.Why is there inconsistent data in facet?**
| The following figure shows the full data statistics results of
Expand All @@ -465,7 +595,8 @@ data more conveniently and extract the specified data fields as follows:
in the above data inconsistency, so cli will use the newer statistical
results.
| |image-20210111111035187|
.. figure:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210111111035187.png
:width: 500px
| **4.Why may the total amount of data in ZoomEye-python and the browser
search the same dork be different?**
Expand All @@ -478,8 +609,11 @@ data more conveniently and extract the specified data fields as follows:
consume more user quota; therefore, in the command line tool, only the
``/host/search`` interface is used for searching.
| |image-20210111141028072|
| |image-20210111141114558|
.. figure:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210111141028072.png
:width: 500px
.. figure:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210111141114558.png
:width: 500px
| **5.The quota information obtained by the info command may be
inconsistent with the browser side?**
Expand All @@ -491,7 +625,7 @@ data more conveniently and extract the specified data fields as follows:
0x07 404StarLink Project
~~~~~~~~~~~~~~~~~~~~~~~~
|image1|
.. figure:: https://github.com/knownsec/404StarLink-Project/raw/master/logo.png
``ZoomEye-python`` is a part of 404Team `Starlink
Project <https://github.com/knownsec/404StarLink-Project>`__. If you
Expand All @@ -510,13 +644,23 @@ partner, you can refer to The way to join the group of Starlink Project.
.. |asciicast| image:: https://asciinema.org/a/qyDaJw9qQc7UjffD04HzMApWa.svg
:target: https://asciinema.org/a/qyDaJw9qQc7UjffD04HzMApWa
.. |image-20210111111035187| image:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210111111035187.png
.. |image-20210111141028072| image:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210111141028072.png
.. |image-20210111141114558| image:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210111141114558.png
.. |image-20210205004653480| image:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210205004653480.png
.. |image-20210205005016399| image:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210205005016399.png
.. |image-20210205004806739| image:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210205004806739.png
.. |image-20210205005117712| image:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210205005117712.png
.. |image-20210205131713799| image:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210205131713799.png
.. |image-20210205131802799| image:: https://raw.githubusercontent.com/knownsec/ZoomEye-python/master/images/image-20210205131802799.png
.. |image1| image:: https://github.com/knownsec/404StarLink-Project/raw/master/logo.png
Loading

0 comments on commit 09fa02b

Please sign in to comment.