index: index update blocks search #13

Closed
opened 10 months ago by René Wagner · 5 comments
Owner

The index update causes the search to fail due to some sort of (exclusive?) lock on the whoosh database or stuff.

The index update causes the search to fail due to some sort of (exclusive?) lock on the whoosh database or stuff.
René Wagner added the
enhancement
label 10 months ago
Poster
Owner

whoosh seems rather unmaintained at the moment, last release was 5 years ago.

Maybe we could dump whoosh and use another FTS engine.
SQLite has a built-in FTS - this could be automatically updated when the crawl data is stored in the db. This means that crawl would automatically mean its indexed and we don't need an additional "index" run.

Not sure how difficult it would be to move the search logic from whoosh to sqlite fts.

This approach would fix #8 & #17 as well.

whoosh seems rather unmaintained at the moment, last release was 5 years ago. Maybe we could dump whoosh and use another FTS engine. SQLite has a built-in FTS - this could be automatically updated when the crawl data is stored in the db. This means that crawl would automatically mean its indexed and we don't need an additional "index" run. Not sure how difficult it would be to move the search logic from whoosh to sqlite fts. This approach would fix #8 & #17 as well.
René Wagner changed title from index update blocks search to index: index update blocks search 9 months ago
Poster
Owner

unfortunately the move to the vps #2 made the situation worse

the vps single-thread performance is way below the high clocked quad-core Xeon which powered the dedicated server before.
Guess i need to tackle this soon

unfortunately the move to the vps #2 made the situation worse the vps single-thread performance is way below the high clocked quad-core Xeon which powered the dedicated server before. Guess i need to tackle this soon
René Wagner self-assigned this 9 months ago
Poster
Owner

I've setup a hacky workaround on geminispace.info which is basically

rm -rf /home/gus/index.old
cp -r /home/gus/index /home/gus/index.new
/home/gus/.poetry/bin/poetry run build_index
mv /home/gus/index /home/gus/index.old
mv /home/gus/index.new /home/gus/index
sudo systemctl restart gus

and a small adjustment in build_index.py to always use the index.new directory ofr updates...unfortunately i messed up the old index during this maintenance ☹️

I've setup a hacky workaround on geminispace.info which is basically ``` rm -rf /home/gus/index.old cp -r /home/gus/index /home/gus/index.new /home/gus/.poetry/bin/poetry run build_index mv /home/gus/index /home/gus/index.old mv /home/gus/index.new /home/gus/index sudo systemctl restart gus ``` and a small adjustment in build_index.py to always use the index.new directory ofr updates...unfortunately i messed up the old index during this maintenance ☹️
Poster
Owner

commit f5ce631246 contains the workaround

commit f5ce631246 contains the workaround
René Wagner removed their assignment 9 months ago
Poster
Owner

nothing more to be done here, will be fixed with the switch to another search provider.

nothing more to be done here, will be fixed with the switch to another search provider.
René Wagner closed this issue 7 months ago
Sign in to join this conversation.
No Milestone
No Assignees
1 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.