Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/akamhy/waybackpy

Wayback Machine API interface & a command-line tool
https://github.com/akamhy/waybackpy

archive-webpage archive-webpages cdx-api internet-archive internet-archiving osint savepagenow wayback-machine wayback-machine-api wayback-machine-python web-archiving webarchiving

Last synced: 29 days ago
JSON representation

Wayback Machine API interface & a command-line tool

Lists

README 

        









Python package & CLI tool that interfaces the Wayback Machine APIs








Unit Tests

codecov

pypi

Downloads

Codacy Badge

GitHub lastest commit

PyPI - Python Version

Code style: black




---



#  Introduction



Waybackpy is a Python package and a CLI tool that interfaces with the Wayback Machine APIs.



Internet Archive's Wayback Machine has 3 useful public APIs.



- SavePageNow or Save API

- CDX Server API

- Availability API



These three APIs can be accessed via the waybackpy either by importing it from a python file/module or from the command-line interface.



##  Installation



**Using [pip](https://en.wikipedia.org/wiki/Pip_(package_manager)), from [PyPI](https://pypi.org/) (recommended)**:



```bash

pip install waybackpy -U

```



**Using [conda](https://en.wikipedia.org/wiki/Conda_(package_manager)), from [conda-forge](https://anaconda.org/conda-forge/waybackpy) (recommended)**:



See also [waybackpy feedstock](https://github.com/conda-forge/waybackpy-feedstock), maintainers are [@rafaelrdealmeida](https://github.com/rafaelrdealmeida/),

 [@labriunesp](https://github.com/labriunesp/)

 and [@akamhy](https://github.com/akamhy/).



```bash

conda install -c conda-forge waybackpy

```



**Install directly from [this git repository](https://github.com/akamhy/waybackpy) (NOT recommended)**:



```bash

pip install git+https://github.com/akamhy/waybackpy.git

```



##  Docker Image



Docker Hub: [hub.docker.com/r/secsi/waybackpy](https://hub.docker.com/r/secsi/waybackpy)



Docker image is automatically updated on every release by [Regulary and Automatically Updated Docker Images](https://github.com/cybersecsi/RAUDI) (RAUDI).



RAUDI is a tool by [SecSI](https://secsi.io), an Italian cybersecurity startup.



##  Usage



### As a Python package



#### Save API aka SavePageNow



```python

>>> from waybackpy import WaybackMachineSaveAPI

>>> url = "https://github.com"

>>> user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0"

>>>

>>> save_api = WaybackMachineSaveAPI(url, user_agent)

>>> save_api.save()

https://web.archive.org/web/20220118125249/https://github.com/

>>> save_api.cached_save

False

>>> save_api.timestamp()

datetime.datetime(2022, 1, 18, 12, 52, 49)

```



#### CDX API aka CDXServerAPI



```python

>>> from waybackpy import WaybackMachineCDXServerAPI

>>> url = "https://google.com"

>>> user_agent = "my new app's user agent"

>>> cdx_api = WaybackMachineCDXServerAPI(url, user_agent)

```

##### oldest

```python

>>> cdx_api.oldest()

com,google)/ 19981111184551 http://google.com:80/ text/html 200 HOQ2TGPYAEQJPNUA6M4SMZ3NGQRBXDZ3 381

>>> oldest = cdx_api.oldest()

>>> oldest

com,google)/ 19981111184551 http://google.com:80/ text/html 200 HOQ2TGPYAEQJPNUA6M4SMZ3NGQRBXDZ3 381

>>> oldest.archive_url

'https://web.archive.org/web/19981111184551/http://google.com:80/'

>>> oldest.original

'http://google.com:80/'

>>> oldest.urlkey

'com,google)/'

>>> oldest.timestamp

'19981111184551'

>>> oldest.datetime_timestamp

datetime.datetime(1998, 11, 11, 18, 45, 51)

>>> oldest.statuscode

'200'

>>> oldest.mimetype

'text/html'

```

##### newest

```python

>>> newest = cdx_api.newest()

>>> newest

com,google)/ 20220217234427 http://@google.com/ text/html 301 Y6PVK4XWOI3BXQEXM5WLLWU5JKUVNSFZ 563

>>> newest.archive_url

'https://web.archive.org/web/20220217234427/http://@google.com/'

>>> newest.timestamp

'20220217234427'

```

##### near

```python

>>> near = cdx_api.near(year=2010, month=10, day=10, hour=10, minute=10)

>>> near.archive_url

'https://web.archive.org/web/20101010101435/http://google.com/'

>>> near

com,google)/ 20101010101435 http://google.com/ text/html 301 Y6PVK4XWOI3BXQEXM5WLLWU5JKUVNSFZ 391

>>> near.timestamp

'20101010101435'

>>> near.timestamp

'20101010101435'

>>> near = cdx_api.near(wayback_machine_timestamp=2008080808)

>>> near.archive_url

'https://web.archive.org/web/20080808051143/http://google.com/'

>>> near = cdx_api.near(unix_timestamp=1286705410)

>>> near

com,google)/ 20101010101435 http://google.com/ text/html 301 Y6PVK4XWOI3BXQEXM5WLLWU5JKUVNSFZ 391

>>> near.archive_url

'https://web.archive.org/web/20101010101435/http://google.com/'

>>>

```

##### snapshots

```python

>>> from waybackpy import WaybackMachineCDXServerAPI

>>> url = "https://pypi.org"

>>> user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0"

>>> cdx = WaybackMachineCDXServerAPI(url, user_agent, start_timestamp=2016, end_timestamp=2017)

>>> for item in cdx.snapshots():

...     print(item.archive_url)

...

https://web.archive.org/web/20160110011047/http://pypi.org/

https://web.archive.org/web/20160305104847/http://pypi.org/

.

. # URLS REDACTED FOR READABILITY

.

https://web.archive.org/web/20171127171549/https://pypi.org/

https://web.archive.org/web/20171206002737/http://pypi.org:80/

```



#### Availability API



It is recommended to not use the availability API due to performance issues. All the methods of availability API interface class, `WaybackMachineAvailabilityAPI`, are also implemented in the CDX server API interface class, `WaybackMachineCDXServerAPI`. Also note

that the `newest()` method of `WaybackMachineAvailabilityAPI` can be more recent than `WaybackMachineCDXServerAPI`'s same method.



```python

>>> from waybackpy import WaybackMachineAvailabilityAPI

>>>

>>> url = "https://google.com"

>>> user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0"

>>>

>>> availability_api = WaybackMachineAvailabilityAPI(url, user_agent)

```

##### oldest

```python

>>> availability_api.oldest()

https://web.archive.org/web/19981111184551/http://google.com:80/

```

##### newest

```python

>>> availability_api.newest()

https://web.archive.org/web/20220118150444/https://www.google.com/

```

##### near

```python

>>> availability_api.near(year=2010, month=10, day=10, hour=10)

https://web.archive.org/web/20101010101708/http://www.google.com/

```



> Documentation is at .



### As a CLI tool



Demo video on [asciinema.org](https://asciinema.org/a/469890), you can copy the text from video:



[![asciicast](https://asciinema.org/a/469890.svg)](https://asciinema.org/a/469890)



> CLI documentation is at .



## CONTRIBUTORS



### AUTHORS



- akamhy ()

- eggplants ()

- danvalen1 ()

- AntiCompositeNumber ()

- rafaelrdealmeida ()

- jonasjancarik ()

- jfinkhaeuser ()



### ACKNOWLEDGEMENTS



- mhmdiaa ()  `--known-urls` is based on [this](https://gist.github.com/mhmdiaa/adf6bff70142e5091792841d4b372050) gist.

- dequeued0 () for reporting bugs and useful feature requests.