index: results to be included
At the moment due to
6eedbd4190/gus/build_index.py (L103) we only index files whose last crawl was a success.
We should be more relaxed in terms of this, which means:
- index files that have a successfull crawl within the last 30 days or the last_crawl was a 20
- we need to filter out redirects, inputs and so on
Maybe we need to adjust the meta data of a page to fullfill this requirements.
- introduce new field
- index fields that have successfull crawl with status 20 in the last 30 days
Deleting a branch is permanent. It CANNOT be undone. Continue?