 Raphael Roberts
|
22d2cc47ab
|
Starting on views and setting up celery settings
|
7 years ago |
|
Raphael Roberts
|
b8d5cb5546
|
Fleshing out some of the functions to get the page, both blocking and async
|
7 years ago |
|
Raphael Roberts
|
624cb73ad6
|
Allow task for PendingScrapingResponse to be null
|
7 years ago |
|
Raphael Roberts
|
ed8757bbf7
|
Fixed some issues with the model fields and added some things
|
7 years ago |
|
Raphael Roberts
|
cfd20b9035
|
Removed unnecessary stuff from __init__.py in scraping sub package.
|
7 years ago |
|
Raphael Roberts
|
90401abe88
|
Moved functions from util.py to models.py because of circular import error
|
7 years ago |
|
Raphael Roberts
|
aff363ba73
|
Added migration to install hstore_extension
|
7 years ago |
|
Raphael Roberts
|
cb92f65ff7
|
Added steps for deployment
|
7 years ago |
|
Raphael Roberts
|
423d600c97
|
Using an idea of promotion. Request -> Pending -> Completed
|
7 years ago |
|
Raphael Roberts
|
88d36e5622
|
Starting on celery tasks and response models
|
7 years ago |
|
Raphael Roberts
|
aa0532a350
|
Made browser_handle for opening new page self.browser_handle
|
7 years ago |
|
Raphael Roberts
|
e8fe97b642
|
Added connect method to browser model and browser cleanup
|
7 years ago |
|
Raphael Roberts
|
e370fa2463
|
Ensured the browser session doesn't auto close when interpreter exits.
|
7 years ago |
|
Raphael Roberts
|
0f59684575
|
Hooking up models to the browser connection
|
7 years ago |
|
Raphael Roberts
|
4eb44250a6
|
Greatly simplified browser.py
|
7 years ago |
|
Raphael Roberts
|
1d49dfa8f5
|
Changed url to be a URLField
|
7 years ago |
|
Raphael Roberts
|
655c4046ce
|
Starting on rest things
|
7 years ago |
|
Raphael Roberts
|
2c25811c60
|
Added encoding to Page model
|
7 years ago |
|
Raphael Roberts
|
de2cbb46b7
|
Blackened codebase
|
7 years ago |
|
Raphael Roberts
|
23aab706d2
|
Using pathlib.Path.parent instead of os.path.join(__fille,".."
|
7 years ago |
|
Raphael Roberts
|
c85da54583
|
Added content_size to Page model
|
7 years ago |
|
Raphael Roberts
|
ab3ee3a901
|
Pascal cased the classes, started on page model, and fixed Browser
class
|
7 years ago |
|
Raphael Roberts
|
102ea770c7
|
Fixed up setup.py and added django files
|
7 years ago |
|
Raphael Roberts
|
fdc370f656
|
Fixed style of scraping components and moved them to submodule 'scraping'
|
7 years ago |
|
Raphael Roberts
|
3d0fde0569
|
started on cache
|
7 years ago |
Raphael Roberts
|
72a1342e54
|
made the param tuple vertical
|
7 years ago |
|
Raphael Roberts
|
1dc31bb7bb
|
Merge branch 'django_app' of https://rlbrhost.ddns.net/git/rlbr/restscrape into django_app
|
7 years ago |
|
Raphael Roberts
|
2ecd077ba3
|
Merge branch 'master' into django_app
|
7 years ago |
|
Raphael Roberts
|
260384397a
|
added a function scrape, which will hopefully be the entry point for everything
|
7 years ago |
|
Raphael Roberts
|
32f94037dd
|
fixed problem with import
|
7 years ago |
|
Raphael Roberts
|
0a19c51f1b
|
Merge branch 'master' into django_app
|
7 years ago |
|
Raphael Roberts
|
3f9ec96359
|
fixed uBlock
|
7 years ago |
|
Raphael Roberts
|
2ae05c9481
|
Merge branch 'master' into django_app
|
7 years ago |
|
Raphael Roberts
|
04bd15e8f7
|
Merge branch 'temp'
|
7 years ago |
|
Raphael Roberts
|
d87baf887d
|
.gitignore ignores everything in uBlock folder
|
7 years ago |
|
Raphael Roberts
|
6a7061d169
|
temporary fix while restoring uBlock
|
7 years ago |
|
Raphael Roberts
|
b3b14ba115
|
Update 'TODO.md'
|
7 years ago |
|
aslkfjkaldsjf
|
3b8338a883
|
starting on django app
|
7 years ago |
|
Gitea
|
78f86cc4ea
|
updated .gitignore to exclude nppBackup
|
7 years ago |
|
Raphael Roberts
|
d00c27ea3b
|
Add 'TODO.md'
|
7 years ago |
|
Raphael Roberts
|
65820cfc17
|
added scraper class to hopefully make things easier
|
7 years ago |
|
Raphael Roberts
|
753eb98914
|
added cache files
|
7 years ago |
|
Raphael Roberts
|
e8afa6c5fc
|
removed browser skeleton from proxy.py
|
7 years ago |
|
Raphael Roberts
|
fb15530eeb
|
added start_page to browser to make getting page quicker
|
7 years ago |
|
Raphael Roberts
|
d7fbd15649
|
added uBlock for blocking ads
|
7 years ago |
|
Raphael Roberts
|
6d1075e6a1
|
added browser class to simplify operations
|
7 years ago |
|
Raphael Roberts
|
ee1a4a4db7
|
renamed page_source.py to proxy.py
|
7 years ago |
|
Raphael Roberts
|
1fb73f2479
|
added proxy server finder
|
7 years ago |
