ah yes, the old 're-implement the search engine inside the database' project, undoubtedly put up on the board because someone is tired of their get-the-data-from-the-database-to-the-search-engine process breaking constantly.
next on the map: discovering how shitty dictionary management is, the joys of NLP, and abandoning the project entirely because you realize all this stuff has already been solved in 3 or 4 different ways and the getting-the-data-from-the-database-to-the-search-engine process isn't really that bad, and pulling your hair out from customers asking insane questions because they don't understand how search engines actually work and why can't this be like google? can't you just do it how google does it, even though you don't have $100B and 50,000 employees?
i realize this is a product feature but i have ptsd on this topic so i had to vent.
the joys of NLP ... this is a search index for Gitlab, that's used primarily to store code, not natural language.
Looking at its competitor, Github's search engine clearly has a base in NLP, and ditches many punctuation characters (https://help.github.com/articles/searching-code/) - which are way more important in code than they are in English - for example, I can't search for code containing "$/" when it should have "\Z/" to match the end of a string. So, right now I have ~250 repos checked out from our Github Enterprise so I can search them offline.
The queries I actually want to run are, pretty much, regexes. Those can be accelerated by trigram indexes like the one gitlab are using (see eg https://swtch.com/~rsc/regexp/regexp4.html), but apparently this isn't implemented by say, Elastisearch (Whereas prefix matching can be made more efficient by preparing your data at index time, wildcard and regular expression matching can be done only at query time.https://www.elastic.co/guide/en/elasticsearch/guide/current/...)
I'd agree with you if this wasn't code. I've seen terrible text search in the DB, and I've built document management systems where we integrated real search engines. But those horses aren't for this course.
sure, primarily code. except when code contains language, like in comments, and other related documentation that's stored in-repo. and of course, people are going to completely mis-use the repo to store ancillary human-readable binary documents, which will need to be converted, indexed, searched, updated, purged, etc, etc...
since developers tag, reference, and document their code with comments, they're going to expect the full suite of NLP treatments with all indexed content, including code.
which basically means: you'll have to tokenize out the special characterize for NLP, but retain them for literal code searches, thereby increasing the size of the inverted index for every permutation of the desired search criteria.
maybe you can detect and filter out all the comments, and index them separately, or maybe have some kind of dual system where the NL indexing system exists separately from the code indexing system ... you see where this is going?
they're going to expect the full suite of NLP treatments with all indexed content, including code. Nope. In 30 years of searching through code, I've never once felt the need for that. On the other hand, almost every time I use Github search, I find the limitations of a word-oriented index get in the way (looking for partial keywords to find code relevant to an api, etc. It's not just special characters).
next on the map: discovering how shitty dictionary management is, the joys of NLP, and abandoning the project entirely because you realize all this stuff has already been solved in 3 or 4 different ways and the getting-the-data-from-the-database-to-the-search-engine process isn't really that bad, and pulling your hair out from customers asking insane questions because they don't understand how search engines actually work and why can't this be like google? can't you just do it how google does it, even though you don't have $100B and 50,000 employees?
i realize this is a product feature but i have ptsd on this topic so i had to vent.