5 replies

Hi Oliver. Thanks for the reply and for listening. I hope things are going really well for you and Memex. 

BOOLEAN:

> Yes this is possible and we have it on the roadmap. So many things compete for our currently, underfunded, very limited attention :( 
But I know it is important and we have it in mind to improve the ability to do AND/OR operators.  

There must be loads going on behind the scenes that people on the outside can't fully appreciate, but i don't really see what could be more fundamental in the bare bones functionality than to have unambiguous control over AND and OR and NOT. This is the niche that the extension Searchmark seemed to do well. Wtih Memex there now seems to be a whole load of functionality being developed around 'organising' rather than 'searching'. Isn't 'organising' what we did with our data in the 20th Century, and now we do 'searching'?

SQL and regex via grep: 

These are just dreams really. The SQL <where>s might refer to fields like 'date of source', 'date added to Memex', 'date last accessed via Memex', 'number of other bookmarks from that domain name', 'only sites from specific domains'..... and regex might be used in the values for those. I'm sure there's loads more you could do with these. Grep, for example, would let you specify how many lines above and below a search term to return. It might be useful to know how close two words appear and with your own control via regex over how those words are stemmed or not. So much that would be so valuable as information overload increases.... I also understand that the more skill required of a tool, then the smaller the pool of customers. But the dumbing down of search tools more generally is about the first vulnerability i've seen in Google's search dominance in 20 years. And i realise that most of this is pie in the sky for now for computational load reasons. 

Maybe better just to get the fundamental requirements of usefulness accomplished first, ie either 'unambiguous Boolean control' OR 'identical search operators to Google' so that the results alongside Google's results are consistent and actually mean something. To go further up the tree of low hanging fruit, to the harder challenges of being able to switch between these two sets of operators, and perhaps also extra search engines, would be much higher benefit to this one user than anything on organising or note taking. 

As no one has infinite time to follow potential future tools, could you please just clarify what Memex is? Is it a search tool? Or is it an organiser and/or note taking tool, maybe like Weava?

Cheers, Jon.

1

Hey Jon, 

Sorry for the ridiculously late reply. 

Unfortunately not yet, as we have been caught up with important other improvements. 
It is on the list of upcoming things, but I can't give an exact ETA, as we are a bit short on (wo)man power at the moment :(

Sorry
- Oliver

Thanks for getting back Oliver.

After being quite excited a few months ago, unfortunately it seems that Memex has moved away from all the things i was looking for in terms of making my information work processes more valuable.

Good luck with it though, Jon.

This is unfortunate to hear! 

What were you excited about? Maybe we move into your desired direction again later, so would be good to understand what you really liked to see?

Thanks for asking Oliver. You might regret it ;-)

The problem i am trying to solve is that when searching the web i need to scan through so much rubbish to get to the good stuff. That quality may be a question of 'enough trust' or 'good insights' or 'good summary with links to other good info' or valuable by some other criterion, like 'entertainment' or 'uptodateness'. The good enough sites get one of their pages bookmarked. All that evaluating sites and pages has taken a lot of time over the years and i waste a lot of time repeating it.

The solution for me would be to be able to search these thousands of bookmarks (not history which includes all the rubbish) before i search wider on the web. I've already evaluated these bookmarked sites and pages, so i'm a lot more confident that what i find in them is going to be worth more. I might find something better out on the whole web but i will have to do a lot more scanning, judging, maybe even checking dates and sources if it's important....

If Google has mainstream web search tied up, there is still space for people (Memex?) to masssively increase the value of our time spent searching. One of those ways is to enable that search through what we've each decided is worth corralling into our bookmarks. At the moment we can all be increasingly worried about what bubble Google (or other engines) are offering us anyway! Is it really the whole web, or some dumbed down perspective on it that they deem we are not capable enough to search ourselves. Is it cos of computation load that we are not offered empowering tools like SQL or regex searching? Can't we be allowed to watch out for comfirmation bias ourselves without a big brother doing it? 

I realise that tools like SQL and grep might be beyond what you want to offer, but a set of operators similar to Google's would seem to be the minimum. This is fundamental to what i thought the tool was supposed to be when i was looking for a replacement to (and immprovement on) Searchmark. Waiting a few seconds cos of algorithmic efficiency, or loads of the other bells and whistles, are way down my list of priorities compared to what i thought was going to be this core competency of 'searching our own, private subset of the web'. But it's not my software, and i love what you're doing, so i'm not complaining. I realise there must be loads of reality going on behind the scenes that we outsiders don't know about and it does seem that these things are part of the goal. I just wish they were the laser focus of it :)

Seeing the results from our corralled, curated and filtered subset of the web (our bookmarks), next to results from a search engine of the whole web is a great idea, if the searches are actually the same, ie the search operators, syntax or whatever is interpreted the same. 

The big thing which i realise is much more difficult is to be able to search the whole site of a bookmark, rather than just that bookmarked page. This would allow another search space between 'the whole WWW' and 'our individually bookmarked pages'. It would effectively be one where we can whitelist or blacklist sites we trust or don't. Rather than 'trust', some people might use criteria which bury their head in the sand and ignore what they don't want to hear but that's their perogative. I still think there's a problem there to be solved in terms of empowering people to overcome information overload in the most valuable way, and that means there's an opportunity. I presume Google are intending to either move into or block that space, presumably watching projects like Memex alongside their own research and dev.

All the best, Jon.

Hey Jon, 

Sorry for the late reply, the last month was packed with so much to do to get Memex being better that I never got around properly answering you. 

Thanks in any case for your extensive documentation about the way you search and the time you took to write this. 

> The solution for me would be to be able to search these thousands of bookmarks (not history which includes all the rubbish) before i search wider on the web

You can already filter by bookmarks and Memex syncs with your browsers bookmarks. 

> SQL searching


What would you expect from a SQL search? How would search look like in your idea?

> regex

Regex is difficult to do and right now technically not possible. Its indeed very hard to do search in the browser. In fact we had to write our own search implementation to afford at least a scalable full text search in the browser. We can't use commonly used server-side technology with all the magic tricks available. 

> I realise that tools like SQL and grep might be beyond what you want to offer, but a set of operators similar to Google's would seem to be the minimum. 

Yes this is possible and we have it on the roadmap. So many things compete for our currently, underfunded, very limited attention :(
But I know it is important and we have it in mind to improve the ability to do AND/OR operators. 

> This would allow another search space between 'the whole WWW' and 'our individually bookmarked pages'.

Actually this was an early idea before Memex: to search selected, trusted, domains. However web-search is expensive to do and also there was not enough demand fo rit. 


Thanks again for all your input :)

Update on search operators?

Hi, is there an updated ETA for boolean search operators please? Thanks, Jon.

1

Just stumbled over this thread again and thought maybe there is also a solution from another angle. 

Did you know, that in the address bar search, if you click on the first result in the list, you jump to the overview where you see all available results?

Have a good weekend :)
Oliver

Also: We recently launched the ability to exclude domains and terms from the search. 
For that just type in: 

Domains:

-site:nytimes.com or -nytimes.com

Terms:

-term1

2

Hey @Princeflickersoul (cool nickname btw) :)

Thanks for the time dropping by and writing down your feedback. Much appreciated. This is how the tool will get better over time!

Right now our search is an AND search, means that only pages that contain all the terms of your query will be shown in the results. 

Example: 
Say you have page 1 with terms house and party, and page 2 with house and pool and page 3 with house, garden and pool.
If you search for each of the words house, party, garden, pool individually, you will get results.
However now if you search 'house party pool' you won't have any results. 

Long story short: 
Yes we know we need to improve on the way we let people query results and it is actually one of the next steps to do better AND/OR queries that also give you results for your use case. 

Will let you know in this thread once we roll out an update.

Have a good weekend.
- Oliver

PS: I changed the title to reflect your needs of improvement. Hope that is fine like that. 
If you feel like it is not capturing the problem right, let me know. 

Thanks! 

It is a great app which really helps me a lot!

Does this ANDing work as you describe? <searches query improve> does not return this page for me, but it surely should do shouldn't it?

Tried both with "Make title and url always.... " switched on and off. 

v0.11.3

@Jon

If the search does not return this page, then there is another bug we need to figure out together. 

For me the page is refindable, so I can't reproduce it. 

In my installation:

<searches query possible> returns this page.

<searches query improve> does not.

<please function keyword> does not.

These suggest that it is just searching the titles doesn't it? This is why in the last post i mentioned about the "Make title and url always searchable...." setting. It is like it is 'Make title and url ONLY searchable.'

Does that help?