index: failure at the end #8

Closed
opened 10 months ago by René Wagner · 6 comments
Owner
2021-02-02 12:25:42,229 index_statistics INFO     Number of type "US-ASCII" charsets indexed: 2
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data/gus/gus/build_index.py", line 132, in main
    build_index(args.should_run_destructive, args.invalidation_window)
  File "/data/gus/gus/build_index.py", line 125, in build_index
    index.close()
  File "/data/gus/gus/lib/search.py", line 75, in close
    self._writer.commit()
  File "/root/.cache/pypoetry/virtualenvs/gus-Nu3YehVz-py3.7/lib/python3.7/site-packages/whoosh/writing.py", line 922, in commit
    finalsegments = self._merge_segments(mergetype, optimize, merge)
  File "/root/.cache/pypoetry/virtualenvs/gus-Nu3YehVz-py3.7/lib/python3.7/site-packages/whoosh/writing.py", line 827, in _merge_segments
    return mergetype(self, self.segments)
  File "/root/.cache/pypoetry/virtualenvs/gus-Nu3YehVz-py3.7/lib/python3.7/site-packages/whoosh/writing.py", line 101, in MERGE_SMALL
    writer.add_reader(reader)
  File "/root/.cache/pypoetry/virtualenvs/gus-Nu3YehVz-py3.7/lib/python3.7/site-packages/whoosh/writing.py", line 710, in add_reader
    self.add_postings_to_pool(reader, basedoc, docmap)
  File "/root/.cache/pypoetry/virtualenvs/gus-Nu3YehVz-py3.7/lib/python3.7/site-packages/whoosh/writing.py", line 647, in add_postings_to_pool
    for item in items:
  File "/root/.cache/pypoetry/virtualenvs/gus-Nu3YehVz-py3.7/lib/python3.7/site-packages/whoosh/writing.py", line 583, in _process_posts
    for fieldname, text, docnum, weight, vbytes in items:
  File "/root/.cache/pypoetry/virtualenvs/gus-Nu3YehVz-py3.7/lib/python3.7/site-packages/whoosh/reading.py", line 429, in iter_postings
    yield (fieldname, btext, m.id(), m.weight(), m.value())
  File "/root/.cache/pypoetry/virtualenvs/gus-Nu3YehVz-py3.7/lib/python3.7/site-packages/whoosh/codec/whoosh3.py", line 1000, in value
    return self._values[self._i]
IndexError: tuple index out of range
``` 2021-02-02 12:25:42,229 index_statistics INFO Number of type "US-ASCII" charsets indexed: 2 Traceback (most recent call last): File "<string>", line 1, in <module> File "/data/gus/gus/build_index.py", line 132, in main build_index(args.should_run_destructive, args.invalidation_window) File "/data/gus/gus/build_index.py", line 125, in build_index index.close() File "/data/gus/gus/lib/search.py", line 75, in close self._writer.commit() File "/root/.cache/pypoetry/virtualenvs/gus-Nu3YehVz-py3.7/lib/python3.7/site-packages/whoosh/writing.py", line 922, in commit finalsegments = self._merge_segments(mergetype, optimize, merge) File "/root/.cache/pypoetry/virtualenvs/gus-Nu3YehVz-py3.7/lib/python3.7/site-packages/whoosh/writing.py", line 827, in _merge_segments return mergetype(self, self.segments) File "/root/.cache/pypoetry/virtualenvs/gus-Nu3YehVz-py3.7/lib/python3.7/site-packages/whoosh/writing.py", line 101, in MERGE_SMALL writer.add_reader(reader) File "/root/.cache/pypoetry/virtualenvs/gus-Nu3YehVz-py3.7/lib/python3.7/site-packages/whoosh/writing.py", line 710, in add_reader self.add_postings_to_pool(reader, basedoc, docmap) File "/root/.cache/pypoetry/virtualenvs/gus-Nu3YehVz-py3.7/lib/python3.7/site-packages/whoosh/writing.py", line 647, in add_postings_to_pool for item in items: File "/root/.cache/pypoetry/virtualenvs/gus-Nu3YehVz-py3.7/lib/python3.7/site-packages/whoosh/writing.py", line 583, in _process_posts for fieldname, text, docnum, weight, vbytes in items: File "/root/.cache/pypoetry/virtualenvs/gus-Nu3YehVz-py3.7/lib/python3.7/site-packages/whoosh/reading.py", line 429, in iter_postings yield (fieldname, btext, m.id(), m.weight(), m.value()) File "/root/.cache/pypoetry/virtualenvs/gus-Nu3YehVz-py3.7/lib/python3.7/site-packages/whoosh/codec/whoosh3.py", line 1000, in value return self._values[self._i] IndexError: tuple index out of range ```
René Wagner added the
bug
label 10 months ago
Poster
Owner

when this happens, the index needs to be completely rebuild using the -d flag.

when this happens, the index needs to be completely rebuild using the `-d` flag.
René Wagner changed title from index updates failes to index updates fails 10 months ago
René Wagner changed title from index updates fails to index: failure at the end 10 months ago
Collaborator

I have run into this multiple times, and even dug into the whoosh code a little bit to try and figure it out - I think it's a bug within whoosh, as I have tried a lot of different index-building settings and approaches that are within our control, and nothing seems to fully eliminate the risk of the error happening.

I have run into this multiple times, and even dug into the whoosh code a little bit to try and figure it out - I *think* it's a bug within whoosh, as I have tried a lot of different index-building settings and approaches that are within our control, and nothing seems to fully eliminate the risk of the error happening.
Poster
Owner

It's back again, seems to happen quite regularly after some time.

It's back again, seems to happen quite regularly after some time.
Poster
Owner

i needed to start a destructive index run and it failed right away with the "tuple index out of range" error at the end.
It seems that this kind of failure leaves some sort of temp files MAIN.tmp in the index folder.

i needed to start a destructive index run and it failed right away with the "tuple index out of range" error at the end. It seems that this kind of failure leaves some sort of temp files `MAIN.tmp` in the index folder.
Poster
Owner

commit f6c3526288 works around the remaining tmp files

commit f6c3526288 works around the remaining tmp files
Poster
Owner

nothing more to be done here

nothing more to be done here
René Wagner closed this issue 7 months ago
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.