1. I agree with this in principal
2. How do we deal with upgrades to J4 when existing sites are using com_search modules and menu items
3. The search statistics page in the admin for com_finder was mistakenly in com_search - this would need to be resolved as its now been deleted

Still waiting for you to modify the stemmer PR...
Also, tests should be done with languages who don't have stemmers and are likely never to get any to see if we should really get rid of com_search.

Note: In any case this PR is incomplete. Missing stuff in script.php and in en-GB.localise.php
plus the other issues described by @brianteeman above

@brianteeman

How do we deal with upgrades to J4 when existing sites are using com_search modules and menu items

I'd say that we should migrate those automatically to com_finder, or is there something that would speak against that?

The search statistics page in the admin for com_finder was mistakenly in com_search - this would need to be resolved as its now been deleted

I'm working on implementing a new search statistics already. That will be another PR in the near future.

@infograf768

Also, tests should be done with languages who don't have stemmers and are likely never to get any to see if we should really get rid of com_search.

Even if there is no stemmer for a language, the results are still several magnitudes better than with com_search. There is no situation where com_search would be equal or even better than com_finder.

Note: In any case this PR is incomplete. Missing stuff in script.php and in en-GB.localise.php

Yes, I missed the stuff in en-GB.localise.php. From what I understand the lines in script.php are script-generated and I wouldn't want to touch that right now.

How do we deal with upgrades to J4 when existing sites are using com_search modules and menu items

Same way the decoupling of weblinks was done. We can remove from repo, the search component/module/plugins should be unprotected so they can be uninstalled from UI, and we need to ensure last 3.x release of com_search will at least function on a 4.0 build. Add a postinstall indicating the component is abandoned/removed and to migrate search platforms.

@mbabker
not exactly same situation specially if the ini files as well as the functions in localise.php are removed.
We must not forget that 4.0 should get totally new lang packs.

You're not going to upgrade from 3.9/10 to 4.0 and all of a sudden the already installed 3.x language packs stop working. So the INI thing is really no concern unless we put in the script.php file "uninstall all the com_search things" code. Which we can't sanely do without breaking sites because for those sites where the component and module are in active use it can and probably will cause problems.

The localise code is a concern that would have to be thought through.

Actually, the localise code only becomes a problem if you remove the corresponding hooks in the Language class. So even if the search code is gone in the en-GB class, if the hooks and default behaviors still exist in Language we'd be OK (at most it alters the search results for the component but it prevents a situation where upgrading the site in full removes something that is user configured somewhere; we don't exactly have a good track record with complex upgrade paths (see 1.5 -> 1.6) so we REALLY need to be careful if someone is going to push for a "well why don't we just delete the component from the database, remove menu items, etc." solution).

So, the part of localise.php which "could" possibly create B/C issues would only be (Here for Chinese Simplified)

	public static function getIgnoredSearchWords()
	{
		return array('啊', '呢', '嘛');
	}

I don't think that we have to do anything special here. 4.0 is a break in B/C and updating to 4.0 includes to switch the search solution to Smart Search.

The statistics for com_finder are now in this PR: #20681

Can we remove the "HOLD" on this PR?

4.0 is a break in B/C and updating to 4.0 includes to switch the search solution to Smart Search.

I do not agree here that the user which updates Joomla must make the switch manually. Joomla has to provide an upgrade path for this pr.

I suggest to wait anyway before taking off com_search as there are some aspects of finder that need special attention, I am thinking specially of the PR that would introduce the length choice for the tuple.
#20384 (comment)

@laoneo as you have established a policy of merging things without them being complete or in some cases even working there is no reason at all for this not to be merged right now and the migration issue addressed in another pr

This PR should only be merged when all other changes to com_finder have been implemented.

@brianteeman Quote from the description of this pr. Not sure how your comment should help bringing this pr further. So please keep the focus and when you want to say something to me, then do it by PM and not in a public tracker. Thanks for understanding.

@laoneo public comments on a public tracker should have public replies. you cant have your cake and eat it.

@laoneo The problem with an automatic upgrade path is, that it is nearly impossible. The user will have to do changes himself, for example checking if every component has a finder plugin that he wants, building the initial index and adapting template overrides from com_search onto com_finder. It wont be enough to simply change the type of module from mod_search to mod_finder and likewise with menu items of com_search. By doing parts of this for the user on upgrade, I fear that we will have lots and lots and lots of users who don't even notice the switch over and thus will not have a (complete) index or all necessary finder plugins.

However, while reviewing the PR just now, we do need indeed some changes. We need a PR that deprecates com_search in 3.9 and also adds a notice that you should prepare to switch over from com_search to com_finder and we need to properly uninstall the search stuff in 4.0.

@infograf768 I do would say that thr finder changes have progressed far enough that we can drop com_search now. The changes that have not been merged yet are very near RTC and the rest of the PRs that I have planned are just small improvements, not major changes like we had in the last few weeks.

@Hackwar then we need also do an announcement as I would say this is a BC break which effects the end user and not extensions. Do you suggest to even recommend to upgrade on J3 first before doing the upgrade to J4, is then a reindex needed after the upgrade?

I do would say that thr finder changes have progressed far enough that we can drop com_search now. The changes that have not been merged yet are very near RTC and the rest of the PRs that I have planned are just small improvements, not major changes like we had in the last few weeks.

@Hackwar
Honestly, it is not so urgent. Let's be safe. If the tuple issue can't be solved and that tuple PR is merged as is, I would oppose strongly to taking off com_search as that one will always work for phrases.

Not considering the consequences of taking off com_search and the way to deal with them right now is also for me a nono to merge this PR now.

Another example is above: #20637 (comment)
If we delete the corresponding methods in the localise.php file while some users still use com_search, they will not be so happy...

@laoneo It would be good if people used com_finder more in 3.x already, simply to also force devs to create plugins for it and not just for com_search. A re-indexing is necessary in any case.

@infograf768 Are you saying that com_finder is not working fine right now? In that case we should delete it completely from the current codebase and first develop this in a separate branch or even a separate project.
com_finder is working fine and has been in production for years. If you are saying that we can not rely on it right now (and thus delete com_search), you are saying that we have been lugging around a huge chunk of code that actually is NOT production ready.
This PR is about removing com_search. If you have issues with another PR, please discuss that there.

As I have already written above, people need to migrate from com_search to com_finder manually. As with the whole migration from 3.x to 4.0, they will have to do manual changes and will not be able to simply do this as an update like they do from 3.8.5 to 3.8.6. I am considering the consequences of this PR and the manual migration is the only thing that will work.

Regarding the changes to localise.php: If the language library would break because localise.php does not have a certain method anymore, then you messed up, not I in this PR. There is no interface for localise.php that would define which methods a translator would have to implement, no base-class to extend from. Rather fix that instead of throwing in another false blocker to prevent these changes from moving forward. If we really want to move forward with Joomla, this is one area where we would simply have to force people to use com_finder. That is why com_search is deleted and not just removed upon new installations.

Even if there is no stemmer for a language, the results are still several magnitudes better than with com_search. There is no situation where com_search would be equal or even better than com_finder.

That's a broad statement which may be true for core extensions. But it certainly isn't true for 3rd party extensions.
You can do things in a search plugin which you can't do in a finder plugin:

• As an example in my extension I have a searchable property of the item (book name of a scripture reference) where you can search for the translated name of the book. In my search plugin I can load the array of those book names and look if the search term is within those names. In Finder, this isn't possible to my knowledge since by the time the item is indexed I don't know the language of the visitor that will run the search in future. Plus language packs may change after the item was indexed anyway.
• With a classic search, the user can also restrict the search to certain areas, and those areas can be created dynamically by the search plugin. To my knowledge, this isn't possible in Finder.
• Last but not least, 3rd party support for com_search is far better than support for com_finder. That's to a degree also because com_finder was not as reliable, and of course also because writing a com_finder plugin is more complex. That will not have changed with the release of 4.0. I'd give more time here for that transition. Assuming Finder is indeed better than Search once all PRs from Hannes are merged, it can be expected that adoption rate of Finder goes up and thus also extension devs will write/improve their respective plugins.

I certainly wouldn't drop Search with 4.0.

Even if there is no stemmer for a language, the results are still several magnitudes better than with com_search. There is no situation where com_search would be equal or even better than com_finder.

That's a broad statement which may be true for core extensions. But it certainly isn't true for 3rd party extensions.
You can do things in a search plugin which you can't do in a finder plugin:

I thought so in the past, but you can do everything with finder that you can do with com_search.

• As an example in my extension I have a searchable property of the item (book name of a scripture reference) where you can search for the translated name of the book. In my search plugin I can load the array of those book names and look if the search term is within those names. In Finder, this isn't possible to my knowledge since by the time the item is indexed I don't know the language of the visitor that will run the search in future. Plus language packs may change after the item was indexed anyway.

I don't understand your example exactly, but I would expect that you would solve that with a search result layout.

• With a classic search, the user can also restrict the search to certain areas, and those areas can be created dynamically by the search plugin. To my knowledge, this isn't possible in Finder.

Finder has a good taxonomy, which allows you to filter far better than with com_search.

• Last but not least, 3rd party support for com_search is far better than support for com_finder. That's to a degree also because com_finder was not as reliable, and of course also because writing a com_finder plugin is more complex. That will not have changed with the release of 4.0. I'd give more time here for that transition. Assuming Finder is indeed better than Search once all PRs from Hannes are merged, it can be expected that adoption rate of Finder goes up and thus also extension devs will write/improve their respective plugins.

3PD had 6 years to adopt com_finder. They did not. Why would that change now? I mean, my changes are bringing good improvements to com_finder, but they are not so revolutionary that a 3PD that ignored com_finder for the last 6 years would now all of a sudden invest work into this.

Remove com_search in 4.0 or it will never be removed.

I am against removing com_search in 4.0, it is a simple solution for search, easy to extend and easy to explain.

Then delete com_finder. There is no reason to keep 10k lines of code in our codebase for the same feature twice and to force responsible 3PD to write and maintain 2 search plugins.

It's not the same feature. Just same (or similar) use case. But they work fundamentally different.

I don't understand your example exactly, but I would expect that you would solve that with a search result layout.

I don't know what a search result layout is. Thus can't say if I can solve my usecase with that.
I can however explain my use case a bit more if you care.
I have my sermon item, this sermon may have n scripture references assign (eg "John 3,16"). That scripture is stored in a separate table, and most importantely for us the book ("John") is stored as a number, not the actual book name. The book name is translated on display using a language string ("COM_SERMONSPEAKER_BOOK_XY).
During search, the user can search for the book name (eg all sermons with references from "John").
In Finder I haven't found a way yet to allow the same. With Search it's pretty simple to do.

Finder has a good taxonomy, which allows you to filter far better than with com_search.

How would a regular website visitor search for sermons with the searchword "sin" from the preacher (not "Author"!) "Billy Sunday"? With Search that is actually simple to do.

Maybe those things are all possible with finder, I honestly don't know. With Search, it's dead simple to achieve.

You can use result-type-specific layouts to render your search results. Add a default_sermon.php template override to your layout and you can modify the output whichever way you want and also translate strings like "John".

com_finder has taxonomy. Install the current 4.0-dev with sample data, index that data in com_finder and then look at "content maps" in the backend or enable advanced search in the frontend. You can filter by category, type, author, price and lots of other stuff. You can also create a filter for that in the backend, create a menu item with that filter and thus create topic specific sub-searches in your site with that filter. One search box could search the whole site, the other only sermons.

There is just one downside: The taxonomy right now does not know about levels, so everything is basically in the same level. But again, that is just something that we have to fix for 4.0. If we don't do this now, we are stuck with com_search (and also the current taxonomy) until 5.0 and that might be another 6 years into the future. Keep in mind that com_search right now is at least 15 years old and has barely changed in that time. And not because it is so "good".

You can use result-type-specific layouts to render your search results. Add a default_sermon.php template override to your layout and you can modify the output whichever way you want and also translate strings like "John".

Learned something today, but that doesn't solve the usecase I have. I don't want to translate something in the result. I "reverse translate" (for lack of a better term) the search term. Eg a german user would search for "Johannes" and it would find the sermons for that book with the ID "23". An english user would search "John" and it also would find the result with that ID "23".

As for taxonomy, I can filter by Author, Category or Type. I have no clue how I could add a "filter by Speaker" dropdown, or even add multiple "Type" entries from the same plugin (which works fine in a Search plugin).

One search box could search the whole site, the other only sermons.

Yeah, but that's not the topic here. It's obvious you can do things with Smart Search which aren't possible with Search. But that goes the other way as well. That's my point.

And not because it is so "good".

Maybe you're wrong with that assumption. Com_search works very well. It's easy to support and is very flexible. You could even write a search plugin which passes the search term into an extern search engine.
Both search components have their use cases. Smart Search has search suggestions and can stem words, but needs to index the content beforehand for that. Classic Search performs a realtime search, thus can't make suggestions but on the other hand doesn't need to do its work beforehand.

Classic Search performs a realtime search, thus can't make suggestions but on the other hand doesn't need to do its work beforehand.

Here's the kicker. Most com_search integrations only support a database query with a WHERE foo LIKE '%bar%' structure, or you're doing some asininely crazy stuff to pull off a result. In a lot of cases, you want your search to expand beyond a literal term match, and that's where com_finder makes its money. Imagine if Google or Bing or Yahoo or Ask Jeeves only supported literal matches.

We really need to pull one of the search engines out of the core distro. Period. As long as search is an entirely plug and play driven system, there is no reason to ship two search components. Pick one for core, make the other "core supported" (and hope it doesn't die like Weblinks, how's that going anyway?). As far as human behavior and human expectations go, I would suggest com_finder more closely aligns with those than com_search does, even with all its flaws and higher technical knowledge to do a correct integration.

We really need to pull one of the search engines out of the core distro.

Why? I don't agree with that statement. It's not like we had a lot of work with those two engines in the past. Especially not with com_search which Hannes himself wrote that we barely had to touch it in the last 15 years.

Pick one for core, make the other "core supported" (and hope it doesn't die like Weblinks, how's that going anyway?).

You can as well delete the one you pull out of core. Because it will be even worse than weblinks (which pretty much died). 3rd party support will drop for the engine that isn't in core and soon after the engine is dead because of no 3rd party support.

I agree that Smart Search usually gives better search results. No arguments here.
But I'd bet most sites still don't use it for whatever reason. I don't use it on my own sites as well as I neither need the better results nor the search suggestions. Classic Search does what I need.

Imho both engines have their use cases. Removing either one would hurt more than what we gain (what do we gain anyway?).

Sorry, wrong button 😄

Even if it’s different it’s duplicated capabilities and I don’t think we
really need both out-of-the-box. Imagine if core shipped com_content and
one of K2, EasyBlog, or the plethora of other content creation components
out there; why should we have two competing solutions in the API?

On Thu, Jun 21, 2018 at 8:20 AM Thomas Hunziker notifications@github.com
wrote:

We really need to pull one of the search engines out of the core distro.

Why? I don't agree with that statement. It's not like we had a lot of work
with those two engines in the past. Especially not with com_search which
Hannes himself wrote that we barely had to touch it in the last 15 years.

Pick one for core, make the other "core supported" (and hope it doesn't
die like Weblinks, how's that going anyway?).

You can as well delete the one you pull out of core. Because it will be
even worse than weblinks (which pretty much died). 3rd party support will
drop for the engine that isn't in core and soon after the engine is dead
because of no 3rd party support.

I agree that Smart Search usually gives better search results. No
arguments here.
But I'd bet most sites still don't use it for whatever reason. I don't use
it on my own sites as well as I neither need the better results nor the
search suggestions. Classic Search does what I need.

Imho both engines have their use cases. Removing either one would hurt
more than what we gain (what do we gain anyway?).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#20637 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAWfobbKS43L7wJsvEm2RlBBAkxg_eHbks5t-52CgaJpZM4UWP6s
.

--

• Michael Please pardon any errors, this message was sent from my iPhone.

why should we have two competing solutions

They're not competing imho. Two different solutions for the same "feature". Both have their areas where they shine.
Content components would be a different thing. They can survive fine as extern solutions. For search engines to survive, they need to have 3rd party support. Whichever you take out of core, will loose that support for sure and will eventually die.

Again, you can do stuff with one that you can do with the other. And again we don't had to put in much work to maintain both.
So to me it looks like we gain nothing by removing it, but loose flexibility.

Whichever you take out of core, will loose that support for sure and will eventually die.

That's a core resource problem. Maintenance of something not bundled in this repo is no different than maintenance of something bundled. At the end of the day we have the same human resources problems regardless of what extensions are bundled versus decoupled.

As for weblinks being dead, I'll let the numbers speak for themselves. Clearly, it is still being installed and updated (used is another discussion, we don't have metrics on that). The numbers tell me though we're doing a disservice to users by not making an effort to maintain that package.

I not the right guy to talk about this, but this is my think:
Let's put it another way. What about if com_finder is the one we propose for put out. We would defend the same criteria, just because com_search is the one we have used all the time? Because is more simple? . IMO com_finder have better future and present than com_search (more, with this recent PRs). And so far, I don't know what you can not do with com_finder, that you do with com_search. But for sure you cant do with com_search, what you can do with com_finder.

Both have their areas where they shine

Not sure about this, where their job is to carry out the same task. IMO, both have same areas, just one try to do the task better than the other.

That's a core resource problem.

No it is not. It's a 3rd party resource problem. 3rd party supports what is in core. 3rd party likely doesn't support "core supported" extensions. A "core supported" com_search would primary die because 3rd party don't write/maintain their plugins anymore, not because com_search would not be maintained.

As for weblinks being dead, I'll let the numbers speak for themselves. Clearly, it is still being installed and updated (used is another discussion, we don't have metrics on that). The numbers tell me though we're doing a disservice to users by not making an effort to maintain that package.

It's dead in regards to not being maintained. Last release (a beta!) is over a year old.

And so far, I don't know what you can not do with com_finder, that you do with com_search.

I've already given examples about what you can do with com_search that you can't do with com_finder.

No it is not. It's a 3rd party resource problem. 3rd party supports what is in core. 3rd party likely doesn't support "core supported" extensions. A "core supported" com_search would primary die because 3rd party don't write/maintain their plugins anymore, not because com_search would not be maintained.

We're doing ourselves a disservice if we're going to flat out say that anything that is not in this repository is abandonware. And if we're really going to take that stance, I've got about 30 repositories with production code for the CMS that need to be merged here so they can be archived.

We continue trying to ship our package as a monolithic system. We continue trying to justify that as "if it's not installed and visible right away then it's dead to everyone". We need to break that mentality, one that only exists because we (the people charged with steering the project's direction) have created it.

Starting using Joomla one of my first Riddles: Whats the difference of "Search" and "Smart Search".

So yes: Reducing this kind of Questions makes using Joomla easier for Newbies.

And if we're really going to take that stance, I've got about 30 repositories with production code for the CMS that need to be merged here so they can be archived.

That's absolutely not what I said.
But God forbid someone is being realistic about the impact of a decision.

I'm out here. I have said what I had to say. No reason to repeat myself here over and over.

The reality isn’t we move something out of core and users stop caring. The
reality is we have something out of core and we stop paying attention to
it. The teams for weblinks and install from web disappeared, yet both are
still heavily used. I don’t agree moving a search package out of core
means the extension ecosystem suddenly abandons things, even optional
features in core don’t get supported by the ecosystem (how many provided
layouts for Hathor or Beez3 or Atomic in the last 2 majors, or to your
point how many support both, or any, search system?).

On Thu, Jun 21, 2018 at 12:10 PM Thomas Hunziker notifications@github.com
wrote:

And if we're really going to take that stance, I've got about 30
repositories with production code for the CMS that need to be merged here
so they can be archived.

That's absolutely not what I said.
But God forbid someone is being realistic about the impact of a decision.

I'm out here. I have said what I had to say. No reason to repeat myself
here over and over.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#20637 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAWfoVud-UL6WsBvz-28HYItDaU792i6ks5t-9ODgaJpZM4UWP6s
.

--

• Michael Please pardon any errors, this message was sent from my iPhone.

For what it is worth, I am all in favor for a single search mechanism in Joomla. Putting in duplicates doesn't make much sense to me.

@Bakual I haven't really done that before, but I just compared the plugin of com_content for com_search and for com_finder with each other and I can say with confidence that writing a plugin for com_search is more complicated than for com_finder.

The plugin for com_finder has 380 lines, quite a lot of which I think we can remove if we invest a bit more time into this for 4.0. The plugin for com_search has 439 lines and you can't remove any of that code. Looking at the code from a complexity perspective, I also think that com_finder wins here. I would say that just roughly 100 lines of code of the finder plugin require me to think and actually write code, the rest is just boilerplate. The com_search plugin in contrast requires a lot more brain power from me for about 380 lines.

The maximum indentation inside a method inside the com_search plugin for com_content is 6. So we are 6 conditions or loops deep. The maximum indentation for the com_finder plugin is 2. I could go on, but I think you get my point. The notion, that com_finder plugins are harder to write than com_search plugins, could be called "Fake News". Sad! 😉

What I have to agree to is, that testing a com_finder plugin is more cumbersome, but I have a script on my todo list to make that easier. That script will execute just that one plugin in a J environment and give some feedback on the indexed data. We can put that into our /build folder for example...

A strong argument for me to switch from com_search to com_finder is, that lots of people simply still follow ancient guides or haven't given search another glance since 1.5. We will not get those people to all of a sudden use com_finder, unless we push them to do that.

And again: 4.0 will be the only chance for us for the next several years to get rid of com_search. If we don't do it now, com_search will be old enough to vote before we have the next chance to remove it and com_finder will be a healthy teenager.

The notion, that com_finder plugins are harder to write than com_search plugins, could be called "Fake News". Sad!

Let's not go down that road, please.

Search plugins are written straight forward. You can read it and understand without having much knowledge about the architecture. Finder on the other hand I have written a plugin, but I can't say I know what each of the properties does. I just copy/pasted a lot of stuff there and it doesn't exactly do what I want.
Complexity isn't just about line statistics and code intendations.

But again (and for last time): I can't do in finder what I do in search and vice versa. Both are mighty tools.

@Bakual There are ways to do in finder what you can do in com_search. But to be honest, if you have such a super special case that it is not covered by com_finder in a reasonable amount of effort, you might be better of by creating your own view for that search.

But again: Ship with one search solution, because it confuses users, because for the enduser it is the same feature and because it wont get more support when 3PD have to support both and because we wont have the chance to remove this from core for years.

My reasons to keep com_search are not related to making specific plugins, but I do understand what @Bakual tries to explain too.

Example:
I have a site specialized in Cancer. That word is used a hundred times (or more) on the site. It is not a huge site though.
Among them is the term (phrase) "Ligue française contre le cancer" which is used a few times only.
When searching for this exact phrase, I do not want to get all the results containing ligue or cancer.

With com_search I get what I want and only what I want (here in one article only) when entering the phrase and setting to "Exact phrase":

With Finder, when using "Ligue française contre le cancer" (with the double quotes) in the search field, except if (after this PR #20384 ) I have set the tuple to 5 (default proposed is 1) which creates an incredible number of rows in the db, I do not get any result when the tuple is set to <5.

Just found another issue in new finder #20384 (comment)

If it doesn't find the exact phrase, then there is a bug in the code, but it doesn't need tuplecount set to 5 to search for a phrase with 5 words.

If it doesn't find the exact phrase, then there is a bug in the code, but it doesn't need tuplecount set to 5 to search for a phrase with 5 words.

In fact if you look at #20384 (comment)
you will see that all these settings just don't work at all.

@infograf768 then there is some more work to do in #20384. However that doesn't have any impact on this decision.

The issue may have been also in the multiple stemmers PR. This is why it is definitely not urgent to take off com_search. All depends (for me) on the final status of com_finder.

And for that it would be nice if people wouldn't run around crying "OMG!! They are trying to change something!! I don't want change! And see? There is a bug! That means everything that is changed is BAD!!!" It would be really helpfull if we had a culture of welcoming positive progress...

hey... Cool down please. It looks like there are a number of people who need some Xanax on this Github project these days. Don't ask me names... 😃

I DO welcome the improvements in Finder. I never used it before because of its limitations.

I have spent more time than anyone testing your finder patches and you have to admit that my suggestions/proposals/bug chasing have been positive until now (as were those from some other people). Alas, few people really tested deeply.

Now, if you consider I have been useless, I will just give up on this.

I did not say that you or your tests were useless.

Having a thought about this topic, I suggest that we kind of deprecate com_search as part of J4, but do not remove it. I guess too many sites use the regular search and the timeframe would be to short for such a hard "BC" break. What we need to do is to make an announcement on j.org as well that com_search is deprecated and that site admins and extension devs should start using and integrating finder.

but do not remove it.

Looks like a reasonable solution.

How long is a reasonable deprecation phase? Why isn't 2 years (last release of 3.x till EOL) enough for that? Why do we have to ship it with the default 4.0 distribution? If people are so dependend on it, then why can't we move it to its own project like with com_weblinks and in worst case let it die slowly? Why does this dinosaur have to be shipped with the Joomla 4.0 default distribution?

Because it is widely used. When you deprecate it with 3.10 and at the same time J4 will be shipped then there is no deprecation phase. Simple as that.

Except nobody is saying flat out abandon it with 4.0. It can be made decoupled, as has been pointed out (and shot down because "reasons") in this thread. Decoupling !== abandonment !== no deprecation phase, if the project is willing to be mature about things and actually manage its resources (which from the sound of things, nobody is willing to commit to maintenance of a decoupled search package, sad panda face).

As can be seen by the statistics for weblinks downloads decoupling doesnt have to mean abandoned. I know I was the one who pointed out that most of those downloads are probably from 2.5 upgrades and people who arent using weblinks and just blindly upgrading. But it does prove without doubt that just because it is decoupled it does not mean there is any b/c break at all There is only a b/c break if coo_search disappeared off the face of the earth

We are deprecating loads of API in 3.8/9/10, which is then removed in 4.0 or changed so much that people have to change their code/sites. Why is that different than this? When we are releasing 4.0, people aren't forced to migrate right away, we have promised them a grace period of 2 years from the last minor release of 3.x or the release of the next major release before we do not provide them with bugfix and security releases. Why do people need more than 2 years? Why do they need 5 or more years to migrate? What do you think would change if people had 5 years compared to 2?

All of that while we could simply NOT delete it from the systems of users when updating to 4.0. The old code should still work for people in 4.0, but new installations wouldn't have this anymore.

Again, I see no reason to keep com_search in the default Joomla 4.0 distribution.

in a perfect world i would simply go with something like elasticsearch
unfortunately we are not in that world

in a perfect world i would simply go with something like elasticsearch
unfortunately we are not in that world

Well, it's also trickier to have core distributed components (stuff in the main ZIP, not necessarily decoupled tools) that have major third party dependencies like that. It's one thing for the database (we need one or we won't function at all the way the app is designed), but a search platform is a bit of a special case.

So, now that all major changes to com_finder have been merged, I'd like to bring this up again.

I understand that com_search is widely used and that we can't really just drop it right away, but I would still like to propose to get active here with J4.0 and decouple com_search from Joomla right now, so that we can manage it as a separate project in the 4.x development cycle and then drop it entirely with 5.0.

I don't expect a lot of developer time necessary to maintain such a project. There are currently 3 issues open for com_search, none of which I see us to address. com_search seems to have run its course in terms of development. The challenge would be to keep it working in 4.x, maybe with some nice testing infrastructure.

I talked to @wilsonge about this and one requirement of him to even consider this was, that there is a maintainer who will care for this project and prevent it from becoming abandonware, at least in the runtime of 4.x. I understand that I have a conflict of interest in that I want to drop com_search in favour of com_finder, but my itch to clean up this double-search-feature-mess is bigger and thus I would volunteer to maintain that project. As I already described, I would not expect that to be a project with loads of new features, but instead to keep it up to date with the development of Joomla and I expect us to drop it in 5.0, but under those conditions, I would be willing to take that responsibility.

In addition I think we need to have some really good documentation on how a developer can write search plugins for finder

Yes, the documentation needs to be improved (it should be better than the one for creating a com_search plugin. That one seems to not have been changed since 1.5...) and we should have some tools to help develop these plugins. I'm working on both.

Decouple it sound like a good plan.

In classmap.php, remove the following?

JLoader::registerAlias('JSearchHelper', '\\Joomla\\CMS\\Helper\\SearchHelper', '5.0');

done.

Delete \administrator\components\com_search\services\provider.php?

Should the deleted files be listed here?
https://github.com/joomla/joomla-cms/blob/4.0-dev/administrator/components/com_admin/script.php#L279

@Quy in an ideal world yes but this hasnt been done in j4 during this alpha stage. It can be done in one go later

Thanks!

Whuah, big step done!

Thanks

Thank you!

didn't we have now the nested tables (#__asset, #__menu) in a not good state ?

Merges like this one are exactly why there should be something similar to https://symfony.com/blog/category/living-on-the-edge on the joomla.org network. Because this major merge isn't going to be documented anywhere except as a footnote of an alpha release (if even that much). And anyone who has listened to me knows I've been saying for years we need that type of proactive blog content and that work on upcoming releases needs to stop being buried in this repository until the stable release goes out when suddenly it's OK to put all the marketing effort behind things (which is way too late).

I agree that we should inform people more and I'm happy to write blogposts for the features that I'm contributing.