Software Request: Open Source Personal Content Repository
Started by thouqht
on 9/11/2018
thouqht
9/11/2018 2:33 am
+++++++++++++++++++++++++++++++++++++
TLDR; I need a "future proof" content database. Am thinking a Wiki or even something like Wordpress. Ease of use, searchability, organization, access, and ability to migrate are significant concerns.
+++++++++++++++++++++++++++++++++++++
Hello CRIMPers,
Perhaps I'm at a point where some of you may have been in the past and am looking for guidance. Here's my use case:
I run a business built upon content marketing and information products. Because of this, I produce a large amount of content ranging from blog posts, social media posts, youtube videos (and their outlines), and various courses.
As of now, this information is stored across a wide range of apps such as dynalist, workflowy, my computer file system, vim applications, evernote (two different accounts), OneNote, google docs, mailing list software, and most recently, notion.so (if only it was open source and self-hosted!).
I'm realizing that as my business progresses and my content output will only increase, this scattered storage has a real opportunity cost. In my line of work, it is very powerful to call up old content for new inspiration, to tweak and repackage, or reference for a larger work like a book.
With my information stored everywhere, it is almost impossible to do that efficiently. At first I was thinking to use Evernote, Notion, or any of the more niche software that this forum is fond of, but with the way things go in the software world, that just seems way too risky.
I would prefer an open source solution with a decent enough following that it could survive past any one person or business deal. And even if it couldn't, it should be of a format that will hopefully be exportable or accessible in the future.
So it got me thinking about other things like wikis. In particular, my current front runners are:
- Tiddlywiki
- Dokuwiki
- Wordpress
So with all that being said, I'd love to hear what thoughts suggestions you all have. Thank you for your feedback.
TLDR; I need a "future proof" content database. Am thinking a Wiki or even something like Wordpress. Ease of use, searchability, organization, access, and ability to migrate are significant concerns.
+++++++++++++++++++++++++++++++++++++
Hello CRIMPers,
Perhaps I'm at a point where some of you may have been in the past and am looking for guidance. Here's my use case:
I run a business built upon content marketing and information products. Because of this, I produce a large amount of content ranging from blog posts, social media posts, youtube videos (and their outlines), and various courses.
As of now, this information is stored across a wide range of apps such as dynalist, workflowy, my computer file system, vim applications, evernote (two different accounts), OneNote, google docs, mailing list software, and most recently, notion.so (if only it was open source and self-hosted!).
I'm realizing that as my business progresses and my content output will only increase, this scattered storage has a real opportunity cost. In my line of work, it is very powerful to call up old content for new inspiration, to tweak and repackage, or reference for a larger work like a book.
With my information stored everywhere, it is almost impossible to do that efficiently. At first I was thinking to use Evernote, Notion, or any of the more niche software that this forum is fond of, but with the way things go in the software world, that just seems way too risky.
I would prefer an open source solution with a decent enough following that it could survive past any one person or business deal. And even if it couldn't, it should be of a format that will hopefully be exportable or accessible in the future.
So it got me thinking about other things like wikis. In particular, my current front runners are:
- Tiddlywiki
- Dokuwiki
- Wordpress
So with all that being said, I'd love to hear what thoughts suggestions you all have. Thank you for your feedback.
Jan S.
9/11/2018 9:21 am
https://orgmode.org/ can do all of that. you might also want to index your files with something like http://www.lesbonscomptes.com/recoll/
Paul Korm
9/11/2018 10:42 am
Jan S's suggestion of org mode is a good example of future proofing -- because it has a long history of having been future proof. It endures.
The key to future proofing your data store is to
1) Make sure your document types are likely to endure and are not proprietary or custom formats. For example, if you understand JSON and do not understand how to transform JSON (or XML) files then do not use apps that create these files. Markdown is often cited as "future proof"
2) Don't store your content solely online in a cloud -- behave and plan as though there is a 100% chance of failure of any given online repository in the course of your career.
3) If you store offline, then refresh your storage media at least once every 24 months. There is also a 100% chance of failure of every storage device.
Wikis are not designed for effective document storage. Your file system is.
Remember the rule 3-2-1 rule: store every document 3 times, in at least 2 locations, 1 of which is off site.
The key to future proofing your data store is to
1) Make sure your document types are likely to endure and are not proprietary or custom formats. For example, if you understand JSON and do not understand how to transform JSON (or XML) files then do not use apps that create these files. Markdown is often cited as "future proof"
2) Don't store your content solely online in a cloud -- behave and plan as though there is a 100% chance of failure of any given online repository in the course of your career.
3) If you store offline, then refresh your storage media at least once every 24 months. There is also a 100% chance of failure of every storage device.
Wikis are not designed for effective document storage. Your file system is.
Remember the rule 3-2-1 rule: store every document 3 times, in at least 2 locations, 1 of which is off site.
Paul Korm
9/11/2018 10:43 am
Meant to write "if you do *not* understand JSON and do *not* understand how to transform..."
thouqht
9/11/2018 1:36 pm
Thanks for the feedback, Paul, I try to stick with markdown or formatted html where I can for sure.
As for the org-mode rec... you might be spot on...
I've tried to use it before as my main editor, but no matter how much I fiddled with spacemacs or evil mode, I just couldn't get it as nice as vim for pure editing and would ultimately always just go BACK to vim (primarily because vim with easy-motion is hands down THE most efficient way to edit text, and the emacs alternatives were always just a little bit lackin).
However, within the context of using it as a content repo, it might indeed be perfect...
After doing some more research, I'm between org-mode and tiddlywiki... both would take some effort to get up and running, but by the nature of this being a long-term project, that's not the biggest deal.
Any thoughts on the pros vs cons between these would be helpful.
As for the org-mode rec... you might be spot on...
I've tried to use it before as my main editor, but no matter how much I fiddled with spacemacs or evil mode, I just couldn't get it as nice as vim for pure editing and would ultimately always just go BACK to vim (primarily because vim with easy-motion is hands down THE most efficient way to edit text, and the emacs alternatives were always just a little bit lackin).
However, within the context of using it as a content repo, it might indeed be perfect...
After doing some more research, I'm between org-mode and tiddlywiki... both would take some effort to get up and running, but by the nature of this being a long-term project, that's not the biggest deal.
Any thoughts on the pros vs cons between these would be helpful.
Glen Coulthard
9/11/2018 2:28 pm
It seems that your use-case is becoming more and more popular these days. As a professor, I'm in a similar situation of content production and curation (and textbook authoring). Either self-produced or downloaded, I am often swimming in PDF, PPTX, XLSX, DOCX, PNGs, mindmaps, and other file formats. I've tried to get a handle on indexing content through my file system using X1 Search and Copernic, but haven't been satisfied with the "discoverability" or "surfacing" of past data.
If it helps, here is what I use as a Windows/Linux user:
1. Dokuwiki - for wiki content that I moved from ConnectedText (oh, how I wish CT was still a going concern!) Wikis are great for text-based content and research links, revision control, searching, and quick editing/updating.
2. Markdown - for note-taking on various devices (Win/Lin/Mobile), synced online through Dropbox, Nextcloud, and/or Simplenote.
3. Cintanotes (Win) - for quick notes, web clipping, tagging, and most of the things I used to do in Evernote.
4. Scrivener (Win) - for serious writing, idea formation, and mapping out blog posts, lectures, and book chapters.
5. MindManager (Win) - for brainstorming, conceptualization, lecture production, and more. (I like the fact that I can attach links and documents to branches and then export out as a Word outline. I also have a lot of content in TheBrain, but have a tough time using it as a daily "trusted home" for my data.)
As a CRIMPer, I still have content stored in traditional outliners like myBase and MyInfo, but haven't used them as much these days. The reason that I like Markdown is that I can easily generate searchable static websites and/or ebooks from the content (see MKDocs or Docusaurus), or I can convert the content into WordPress posts. I also host my own home-based unRAID server with GitLab (for markdown docs and revision-control), Nextcloud, Dokuwiki, and document shares - this lets me access the same content from any desktop, laptop, tablet, and/or phone (both within my home network and externally through a VPN.) Much of this server data is also synced with the cloud for backing up...but that's another post
Anyway, hope that helps.
Glen
If it helps, here is what I use as a Windows/Linux user:
1. Dokuwiki - for wiki content that I moved from ConnectedText (oh, how I wish CT was still a going concern!) Wikis are great for text-based content and research links, revision control, searching, and quick editing/updating.
2. Markdown - for note-taking on various devices (Win/Lin/Mobile), synced online through Dropbox, Nextcloud, and/or Simplenote.
3. Cintanotes (Win) - for quick notes, web clipping, tagging, and most of the things I used to do in Evernote.
4. Scrivener (Win) - for serious writing, idea formation, and mapping out blog posts, lectures, and book chapters.
5. MindManager (Win) - for brainstorming, conceptualization, lecture production, and more. (I like the fact that I can attach links and documents to branches and then export out as a Word outline. I also have a lot of content in TheBrain, but have a tough time using it as a daily "trusted home" for my data.)
As a CRIMPer, I still have content stored in traditional outliners like myBase and MyInfo, but haven't used them as much these days. The reason that I like Markdown is that I can easily generate searchable static websites and/or ebooks from the content (see MKDocs or Docusaurus), or I can convert the content into WordPress posts. I also host my own home-based unRAID server with GitLab (for markdown docs and revision-control), Nextcloud, Dokuwiki, and document shares - this lets me access the same content from any desktop, laptop, tablet, and/or phone (both within my home network and externally through a VPN.) Much of this server data is also synced with the cloud for backing up...but that's another post
Anyway, hope that helps.
Glen
nathanb
9/11/2018 3:23 pm
Based on your finalists, it looks like you are looking for a front-end index to a bunch of mixed media files (photos, vids, pdfs etc)? That you are planning on storing the media in a file system archive, not 'inside' the database of the index system?
Tagspaces is probably what I'd use if I wanted a 'future proof' index of a file collection. It also does light note-taking as a front-end to plain-text markdown files.
Tagspaces is probably what I'd use if I wanted a 'future proof' index of a file collection. It also does light note-taking as a front-end to plain-text markdown files.
thouqht
9/11/2018 5:10 pm
nathanb wrote:
Based on your finalists, it looks like you are looking for a front-end
index to a bunch of mixed media files (photos, vids, pdfs etc)? That
you are planning on storing the media in a file system archive, not
'inside' the database of the index system?
Tagspaces is probably what I'd use if I wanted a 'future proof' index of
a file collection. It also does light note-taking as a front-end to
plain-text markdown files.
Hmmm that's an interesting thought... I think as purely a front end to my file system that would work great. However as you suggest this, it makes me realize that I'd like to be able to create links *between* files as well. This is something that tiddlywiki, org-mode, or dokuwiki do quite well. However, perhaps a sophisticated enough tagging system would accomplish similar results...
MadaboutDana
9/11/2018 8:11 pm
@Glen,
Yes, I think Dokuwiki is one of those unsung giants that's actually far more powerful than it appears at first sight. Interesting insights into your software collection (and the reasoning for it, where not, of course, driven by CRIMPing ;-)) - thanks!
Cheers,
Bill
Yes, I think Dokuwiki is one of those unsung giants that's actually far more powerful than it appears at first sight. Interesting insights into your software collection (and the reasoning for it, where not, of course, driven by CRIMPing ;-)) - thanks!
Cheers,
Bill
Alexander Deliyannis
9/12/2018 7:15 pm
Glen Coulthard wrote:
I agree about the 'popularity' of this situation. I would note three main requirements:
- Easy collection of material, either one's own or from other sources, wherever one may be: via browser, email, clipboard, file system, open document, even on the field with a smartphone.
- Full search of that material, including files, plus ability to organise it via tags or folders etc.
- Ability to clip, edit, clone, recycle, and otherwise process the material, and publish it in a variety of media.
I very much appreciate Open Source Software, and in terms of longevity, nothing beats plain text--see Project Gutenberg as an example. Thereon however, I have yet to find an OSS which excels at the third bullet, and this is crucial in my case. My team chose Atlassian Confluence for a major knowledge management project and the result has vindicated us. This is not to say that it is without its flaws.
The article below, which I believe has been cited again in this forum, describes such a content management application with Confluence as its central tool:
https://contentmarketinginstitute.com/2015/02/technology-behind-language-of-content-strategy/
It seems that your use-case is becoming more and more popular these
days. As a professor, I'm in a similar situation of content production
and curation (and textbook authoring).
I agree about the 'popularity' of this situation. I would note three main requirements:
- Easy collection of material, either one's own or from other sources, wherever one may be: via browser, email, clipboard, file system, open document, even on the field with a smartphone.
- Full search of that material, including files, plus ability to organise it via tags or folders etc.
- Ability to clip, edit, clone, recycle, and otherwise process the material, and publish it in a variety of media.
I very much appreciate Open Source Software, and in terms of longevity, nothing beats plain text--see Project Gutenberg as an example. Thereon however, I have yet to find an OSS which excels at the third bullet, and this is crucial in my case. My team chose Atlassian Confluence for a major knowledge management project and the result has vindicated us. This is not to say that it is without its flaws.
The article below, which I believe has been cited again in this forum, describes such a content management application with Confluence as its central tool:
https://contentmarketinginstitute.com/2015/02/technology-behind-language-of-content-strategy/
nathanb
9/12/2018 7:24 pm
thouqht wrote:
I feel your pain. Every time I get excited about links (especially to files), I immediately run into the inherent fragility of one-way linking, especially between different systems. It's dependent on the names and locations staying the same. I think this problem is particularly annoying for file directories since you only get ONE hierarchy scheme and you can't re-arrange it if your future self questions now self's opinions on how stuff should be organized.
After years of CRIMPing and coming up short with a robust system that handles inevitable evolution of organization concepts (and a carousel of trying out different systems), the ONE method I've tried that protects against link fragility is the humble Zettel Code. Well, I call it my Zettel code, but I don't actually follow the Zellkasten system. I just like the concept of a unique text string AS the link.
So now, when I'm working within my task manager, notes, reports etc and want to reference a file. I just generate a new Z-code, which for me is a timestamp to ensure that it's unique unless I live until September 12th, 2119, in which case I hope we've solved linking. So the code looks like 'Z18090121351. It's auto-generated via hotkey and I use military time to ensure uniqueness to this particular minute. The timestamp-as-information is only coincidental as I frequently am linking 'old' stuff. It might tell me what order I ordered stuff in but has no actual bearing on the timeliness of the content it is flagging. I copy this code into the end of the filename/foldername/whatever-content I'm linking to.
Now I only have to follow ONE cardinal rule within my hodgepodge of a digital nightmare of ever changing concepts and rules. That I NEVER change any z-code I run across and I ALWAYS accompany every intra-system reference with a z-code. Beyond that, I'm free to rename/re-arrange as much as I want because I'll always be able to find that reference via basic search. This stupid simple method has made all the difference and freed me from trying to establish THE perfect system and instead slowly evolve a combination of loosely connected parts.
It's pretty cool to be able to reference my 2018 tax return pdf from within my notes, task manager, simple documents, calendar, email etc and that 'link' will still work under all the following circumstances:
-I 'migrate' all my files from OneDrive to dropbox/gdocs/NAS etc.
-I completely re-arrange all my personal files, both names (except the z-code part) and folder structure
-I change to a different note system/wiki/task manager etc.
The worst case scenario is having to search more than two places for the reference. The other downside is that all my systems and directories and names are uglied-up by these z-codes now. So aesthetically it's annoying but it is a constant reminder of how robust my links are AND it's a good flag that something is important enough to have been referenced from another thing.
This method has been the ONE constant in my digital life for about two years now and I continue to find new uses for it. I started doing this specifically for reference documents and now it is my go-to solution for a lot more. Then I started using it to flag tasks and projects. I now have my own subcategories of z-codes. ZM*** is a meeting and therefore a hint to search my calendars first for it. ZT*** is a task so I'd look first inside my task manager for the matching code. ZP*** means project which means it probably has it's own folder in one of 3 different file systems and probably it's own section in OneNote/Notion and maybe even it's own Outlook folder if it's for work. As I get more granular with it, I'm now to the point where I'm approaching true Zettelkasten territory, where each z-code represents a much more granular bit of info like idea, goal, category, quote, book etc... but honestly I'm not about to give up on more appropriate systems for those things by applying a silly plain text solution to all of it. I'm not ready to admit CRIMPing defeat and revert to just text files. But these Z-codes have been a super reliable life-raft that lets me play around with different components without interrupting my workflow.
nathanb wrote:
Based on your finalists, it looks like you are looking for a front-end
>index to a bunch of mixed media files (photos, vids, pdfs etc)? That
>you are planning on storing the media in a file system archive, not
>'inside' the database of the index system?
>
>Tagspaces is probably what I'd use if I wanted a 'future proof' index
of
>a file collection. It also does light note-taking as a front-end to
>plain-text markdown files.
Hmmm that's an interesting thought... I think as purely a front end to
my file system that would work great. However as you suggest this, it
makes me realize that I'd like to be able to create links *between*
files as well. This is something that tiddlywiki, org-mode, or dokuwiki
do quite well. However, perhaps a sophisticated enough tagging system
would accomplish similar results...
I feel your pain. Every time I get excited about links (especially to files), I immediately run into the inherent fragility of one-way linking, especially between different systems. It's dependent on the names and locations staying the same. I think this problem is particularly annoying for file directories since you only get ONE hierarchy scheme and you can't re-arrange it if your future self questions now self's opinions on how stuff should be organized.
After years of CRIMPing and coming up short with a robust system that handles inevitable evolution of organization concepts (and a carousel of trying out different systems), the ONE method I've tried that protects against link fragility is the humble Zettel Code. Well, I call it my Zettel code, but I don't actually follow the Zellkasten system. I just like the concept of a unique text string AS the link.
So now, when I'm working within my task manager, notes, reports etc and want to reference a file. I just generate a new Z-code, which for me is a timestamp to ensure that it's unique unless I live until September 12th, 2119, in which case I hope we've solved linking. So the code looks like 'Z18090121351. It's auto-generated via hotkey and I use military time to ensure uniqueness to this particular minute. The timestamp-as-information is only coincidental as I frequently am linking 'old' stuff. It might tell me what order I ordered stuff in but has no actual bearing on the timeliness of the content it is flagging. I copy this code into the end of the filename/foldername/whatever-content I'm linking to.
Now I only have to follow ONE cardinal rule within my hodgepodge of a digital nightmare of ever changing concepts and rules. That I NEVER change any z-code I run across and I ALWAYS accompany every intra-system reference with a z-code. Beyond that, I'm free to rename/re-arrange as much as I want because I'll always be able to find that reference via basic search. This stupid simple method has made all the difference and freed me from trying to establish THE perfect system and instead slowly evolve a combination of loosely connected parts.
It's pretty cool to be able to reference my 2018 tax return pdf from within my notes, task manager, simple documents, calendar, email etc and that 'link' will still work under all the following circumstances:
-I 'migrate' all my files from OneDrive to dropbox/gdocs/NAS etc.
-I completely re-arrange all my personal files, both names (except the z-code part) and folder structure
-I change to a different note system/wiki/task manager etc.
The worst case scenario is having to search more than two places for the reference. The other downside is that all my systems and directories and names are uglied-up by these z-codes now. So aesthetically it's annoying but it is a constant reminder of how robust my links are AND it's a good flag that something is important enough to have been referenced from another thing.
This method has been the ONE constant in my digital life for about two years now and I continue to find new uses for it. I started doing this specifically for reference documents and now it is my go-to solution for a lot more. Then I started using it to flag tasks and projects. I now have my own subcategories of z-codes. ZM*** is a meeting and therefore a hint to search my calendars first for it. ZT*** is a task so I'd look first inside my task manager for the matching code. ZP*** means project which means it probably has it's own folder in one of 3 different file systems and probably it's own section in OneNote/Notion and maybe even it's own Outlook folder if it's for work. As I get more granular with it, I'm now to the point where I'm approaching true Zettelkasten territory, where each z-code represents a much more granular bit of info like idea, goal, category, quote, book etc... but honestly I'm not about to give up on more appropriate systems for those things by applying a silly plain text solution to all of it. I'm not ready to admit CRIMPing defeat and revert to just text files. But these Z-codes have been a super reliable life-raft that lets me play around with different components without interrupting my workflow.
Franz Grieser
9/12/2018 8:37 pm
Thanks nathanb for the explanation.
What I am not sure I really understand: You have a file named (let's say) "outline for sci-fi novel.doc" and reference this file in 3 other places. Then you'd have to add 3 times a code like "Z18090121351" to the file name. Right? So, depending on the file system you're using, there is a limit to the number of references you can add to the file name. In Windows a file name including the folder path may not be longer than 255 digits (ok, there seems to be a registry hack that permits longer names but I haven't tested that). What do you do when you reach this limit?
What I am not sure I really understand: You have a file named (let's say) "outline for sci-fi novel.doc" and reference this file in 3 other places. Then you'd have to add 3 times a code like "Z18090121351" to the file name. Right? So, depending on the file system you're using, there is a limit to the number of references you can add to the file name. In Windows a file name including the folder path may not be longer than 255 digits (ok, there seems to be a registry hack that permits longer names but I haven't tested that). What do you do when you reach this limit?
nathanb
9/12/2018 9:43 pm
Franz Grieser wrote:
Thanks nathanb for the explanation.
What I am not sure I really understand: You have a file named (let's
say) "outline for sci-fi novel.doc" and reference this file in 3 other
places. Then you'd have to add 3 times a code like "Z18090121351" to the
file name. Right? So, depending on the file system you're using, there
is a limit to the number of references you can add to the file name. In
Windows a file name including the folder path may not be longer than 255
digits (ok, there seems to be a registry hack that permits longer names
but I haven't tested that). What do you do when you reach this limit?
Sorry I wasn't clear. I don't have any filenames with more than one z-code. Maybe it'd be more accurate to say that the z-code is a special tracker for that file instead of the code representing a particular link between two things. In the case of only one note referring to only one file, the distinction is unnecessary.
So if I have several notes/projects/tasks etc referring to the same file, it's going to be with the same Z-code they all share.
You bring up a pretty good point though about the difference between two-way linking (otherwise called directed graph, transclusion, back-links etc) and simple one-way 'blind' linking. I'm forever trying to figure out the best systems with true two-way links and we can only ever achieve that within a single database/wiki/app because it requires that the link itself (the relationship) be it's own animal with its own properties whereas classic 'blind' links are nothing more than a road-sign with an address that may or may not still exist.
The way I'm using Z-codes is simply to make blind-linking more robust. I guess I never really thought about the codes representing an instance of a relationship between two things, which I suppose is closer to how a true Zettelkasten system works. That WOULD result in many additional codes assigned to the more popular targets as you describe. In that case I would start to just embed them within the files themselves.
I honestly haven't ran up against that because my reference links aren't nearly as intricate/circular as they'd be if I were trying to write a thesis or something. In that case I'd be trying to contain all the link relationships within software that handles two-way linking automatically.
Simon
9/13/2018 7:08 am
Similar to many of you, I've used many many many applications and workflows. What has stuck for me is simplicity. If the system isn't simple, you won't use it. Similar to nathanb I use a date stamp at the front of all documents. So for example a leaders agenda becomes 20180913-leaders agenda.txt. I also use tags in the name so a leaders agenda for my organisation (Hope Church) would be, 20180913-hc-leaders agenda.txt. I tend to write titles from the perspective of what will I look for in the future.
On the Mac Devonthink is my main repository for everything and has been for a decade. It has gigabytes of data strewn across 12 databases and I can pretty much find anything. Plus, all data can be exported so no lock in. I have 70K emails in Devonthink and it doesn't blink and finds 10 year old emails with great ease. Plus the ability to configure the way the data is displayed is immense. I have smart folders set so that in my emails archive database there is a smart folder for each year from 2007 to 2018. This is just the beginning.
I also have a Finder repository that was started back in 2007 with AmberV's comment from Literature & Latte linked in an earlier post in this thread. This is a simple date based repository. The folders all follow the pattern 2017 > qtr1; 2017 > qtr2, and so forth. I have a 30 day folder in which all my current work is saved. If a particular document is not modified in 30 days a background script adds the date/time stamp to the beginning of the document and files it in the relevant year and quarter. As I use area tags and descriptive titles I tend to be able to find most things in the system within 30 seconds going 11 years back when I started. It took some steeling of the nerves when I first started because up-un-till then everything was stored in subject based folders and dismantling that produced a large amount of fear. However, I'm totally happy with that system. Hats off to AmberV! I hardly ever file anything and as the script does everything and I only work from my 30 day folder period.
The quest to have everything in one place has led me to using many different programmes and methods and has in fact had the opposite effect of having many things in many places. I have come to the conclusion that my main problem is that I don't stick with something long enough. The longer I stick with a workflow or system the more useful it becomes. The constant changing of systems does not allow my workflow to become mature. My greatest success in this area has come with using Workflowy for all my text based information. If I need something I wrote down it will be in Workflowy and finding hasn't yet proved to be a problem, plus the tagging system is extremely helpful. I'm also beginning to experiment with Ulysses. I'm at a stage where I can see a clear distinction between my note taking, meeting notes, snippets of information and my research, teaching and preaching needs. I'm using Ulysses exclusively for preaching, teaching and research. This is working well as one of Ulysses' strong points in my opinion is that it emulates the Silver searcher (Ag) found in a plugin in emacs. This means that Ulysses; when I search; not only displays the hierarchy of where the found item is located, but it also shows the line displaying the text snippet of my search criteria. Should the same search criteria be in more than on place in a single sheet of Ulysses, each line is shown. This really allows me to find a needle in a haystack.
My advice would be to find something that works and discipline yourself to stick with it for at least 6 months. Then you can begin tweaking.
On the Mac Devonthink is my main repository for everything and has been for a decade. It has gigabytes of data strewn across 12 databases and I can pretty much find anything. Plus, all data can be exported so no lock in. I have 70K emails in Devonthink and it doesn't blink and finds 10 year old emails with great ease. Plus the ability to configure the way the data is displayed is immense. I have smart folders set so that in my emails archive database there is a smart folder for each year from 2007 to 2018. This is just the beginning.
I also have a Finder repository that was started back in 2007 with AmberV's comment from Literature & Latte linked in an earlier post in this thread. This is a simple date based repository. The folders all follow the pattern 2017 > qtr1; 2017 > qtr2, and so forth. I have a 30 day folder in which all my current work is saved. If a particular document is not modified in 30 days a background script adds the date/time stamp to the beginning of the document and files it in the relevant year and quarter. As I use area tags and descriptive titles I tend to be able to find most things in the system within 30 seconds going 11 years back when I started. It took some steeling of the nerves when I first started because up-un-till then everything was stored in subject based folders and dismantling that produced a large amount of fear. However, I'm totally happy with that system. Hats off to AmberV! I hardly ever file anything and as the script does everything and I only work from my 30 day folder period.
The quest to have everything in one place has led me to using many different programmes and methods and has in fact had the opposite effect of having many things in many places. I have come to the conclusion that my main problem is that I don't stick with something long enough. The longer I stick with a workflow or system the more useful it becomes. The constant changing of systems does not allow my workflow to become mature. My greatest success in this area has come with using Workflowy for all my text based information. If I need something I wrote down it will be in Workflowy and finding hasn't yet proved to be a problem, plus the tagging system is extremely helpful. I'm also beginning to experiment with Ulysses. I'm at a stage where I can see a clear distinction between my note taking, meeting notes, snippets of information and my research, teaching and preaching needs. I'm using Ulysses exclusively for preaching, teaching and research. This is working well as one of Ulysses' strong points in my opinion is that it emulates the Silver searcher (Ag) found in a plugin in emacs. This means that Ulysses; when I search; not only displays the hierarchy of where the found item is located, but it also shows the line displaying the text snippet of my search criteria. Should the same search criteria be in more than on place in a single sheet of Ulysses, each line is shown. This really allows me to find a needle in a haystack.
My advice would be to find something that works and discipline yourself to stick with it for at least 6 months. Then you can begin tweaking.
Franz Grieser
9/13/2018 12:20 pm
nathanb wrote:
I see. Thanks.
You also mentioned TagSpaces in the thread. Do you (still) use it? I find the concept appealing but wouldn't want TagSpaces to mess with my files' names. I know that the Pro edition uses a different mechanism but the free trial doesn't.
Sorry I wasn't clear. I don't have any filenames with more than one
z-code. Maybe it'd be more accurate to say that the z-code is a special
tracker for that file instead of the code representing a particular link
between two things. In the case of only one note referring to only one
file, the distinction is unnecessary.
So if I have several notes/projects/tasks etc referring to the same
file, it's going to be with the same Z-code they all share.
I see. Thanks.
You also mentioned TagSpaces in the thread. Do you (still) use it? I find the concept appealing but wouldn't want TagSpaces to mess with my files' names. I know that the Pro edition uses a different mechanism but the free trial doesn't.
Franz Grieser
9/13/2018 12:23 pm
Re: TagSpaces
My fault: The trial supports the sidecar files for tagging without interfering with the file names.
I know that the Pro edition uses a different mechanism but
the free trial doesn't.
My fault: The trial supports the sidecar files for tagging without interfering with the file names.
Amontillado
9/13/2018 1:13 pm
I use a timestamp prefix on file names, particularly in my personal records/bookkeeping DevonThink database, but I've gotten on a plain text kick (again) lately, particularly for documentation and creative writing. Of course, I repeat myself. In the marketing department, documentation is creative writing.
DevonThink is a hard-to-break habit. It keeps files as files, so if DT vaporizes the files aren't lost. Tags and replications, maybe, but not the files themselves, and I've made an interesting discovery.
Imagine a novel in a number of separate files in a group, comprising the chapters and scenes in your work.
If you set a group to "unsorted," you can drag and drop the order of files in the group. Conveniently, the order you set is persistent. You can play with a manual sort, switch to sort by name (or any criteria), and then when you switch back to unsorted you get your manual sort again. Newly added files go at the end of your manual sort.
So, a collection of chapter and scene files can be safely sorted into the correct order. "Compiling," as Scrivener-speak would have it, works pretty easily with Pandoc or something similar.
Pick a view that shows the folder list and the files within folders and switch to unsorted to get your custom order. Click on a file and use Command-A to select them all. Hit Command-C to copy them.
Now bring up a terminal window. When you hit Command-V at the command line prompt, you get the list of full path names to all your files, with spaces properly backslash-escaped.
To compile a bunch of markdown files in the right order to docx: pandoc -s -o gatsby.docx -f markdown -t docx Command-V
Pretty cool. Those pandoc options are -s (create complete file, not a fragment), -o (output to gatsby.docx), -f (from markdown syntax) -t (to docx format). Command-V pasted the list of source files into the command line. Hitting enter did the rest.
A pure plain text solution would be nice. I kind of like DT's replicants, though. Like symlinks without worry of which is the file and which is the reference.
DevonThink is a hard-to-break habit. It keeps files as files, so if DT vaporizes the files aren't lost. Tags and replications, maybe, but not the files themselves, and I've made an interesting discovery.
Imagine a novel in a number of separate files in a group, comprising the chapters and scenes in your work.
If you set a group to "unsorted," you can drag and drop the order of files in the group. Conveniently, the order you set is persistent. You can play with a manual sort, switch to sort by name (or any criteria), and then when you switch back to unsorted you get your manual sort again. Newly added files go at the end of your manual sort.
So, a collection of chapter and scene files can be safely sorted into the correct order. "Compiling," as Scrivener-speak would have it, works pretty easily with Pandoc or something similar.
Pick a view that shows the folder list and the files within folders and switch to unsorted to get your custom order. Click on a file and use Command-A to select them all. Hit Command-C to copy them.
Now bring up a terminal window. When you hit Command-V at the command line prompt, you get the list of full path names to all your files, with spaces properly backslash-escaped.
To compile a bunch of markdown files in the right order to docx: pandoc -s -o gatsby.docx -f markdown -t docx Command-V
Pretty cool. Those pandoc options are -s (create complete file, not a fragment), -o (output to gatsby.docx), -f (from markdown syntax) -t (to docx format). Command-V pasted the list of source files into the command line. Hitting enter did the rest.
A pure plain text solution would be nice. I kind of like DT's replicants, though. Like symlinks without worry of which is the file and which is the reference.
nathanb
9/13/2018 3:59 pm
Simon wrote:
>I have come to the conclusion that my main problem is that I don't stick with something long enough.
The rest of your post tends to disagree with you! I get what you are saying though. Your core Devonthink method is impressive and I'm jealous and impressed by your archive. This is my favorite kind of description, of what someone has actually done for years. That's what separates concepts, of which all of us here are drawn to regardless, and viable action. Thank you for your post. I've found it helpful.
nathanb
9/13/2018 4:52 pm
Franz Grieser wrote:
You also mentioned TagSpaces in the thread. Do you (still) use it? I
find the concept appealing but wouldn't want TagSpaces to mess with my
files' names. I know that the Pro edition uses a different mechanism but
the free trial doesn't.
Ha! No I don't actually use it. I've tested it out a few times and it is one of my favorite options to possibly transition to. I like it's open nature and because it embeds everything in the filename you don't have to worry about a particular installation breaking or being tied to one workstation. I wasn't put off by it extending filenames, I'm quite used to butchering filenames with Z-codes anyway and actually like that seeing intentional tagging/z-coding following the actual name is itself information that tells me that file is special enough in some way to have warranted this extra forced layer of metadata. I mean, we have tens of thousands of files littering our hard-drives...er...storage mediums. Only of which a few hundred/thousand enough are personally relevant enough to be ongoing 'reference' material. I'm going to want to add some kind of marker to differentiate the signal from the noise.
The reason I don't use tagspaces is because my zettel-stamping of some of my files/folders has proven 'good enough' organization. This is mostly because I've given up on straightening out my directories and just consider that a 'back-end' to my info system. Like 3% of my files are ones I seek out and open directly on a regular basis by drilling down into the folders. Basically my current hot projects. I rely on my note systems (currently OneNote and Notion) and task manager (currently todoist) to be my index to most my stuff.
There is of course a lot of value in having a well-organized file system like some other people here have described (my hat's-off to them!). It's been on my to-do list for years now. I've started it like 30 times and instantly run into roadblocks of deciding whether this tax document goes under 'financial', '2017', 'archives' etc. Or if this manual to my mower should be associated with the receipt and therefore by month bought or in some 'house-maintenance' folder. So I just think in circles for a few minutes, get frustrated at the inherent limitations of single-hierarchy filing, give up and do something else. If I ever decide to finally suck it up and spend an entire weekend whipping my files in shape, it's definitely going to be with tagspaces...because that's the only tool I'm aware of that allows me to organize via multiple categories and doesn't lock me into a proprietary system.
I don't like the idea of relying on simple search in lieu of intentional organizing, but that's my reality. However, it turns out that blind search and my scattered z-codes have proven to be reliable. I can usually find what I need, when I need it. So I don't know if it'd be worth the efficiency/clarity gain to intentionally tag/file ALL of it and keep it that way. I'm starting to come to peace with most of my digital mess remaining a mess, with the occasional z-code ensuring findability of the neat stuff. Then selectively putting structure on a select few subsets of things. I'm finding Notion.so to be quite helpful with that. It lets me dump-notes away in my normal randomish One-Note fashion and occasionally embed real databases to add structure on-demand. That's perfect for things like reading lists, product research, software research etc.
Donovan
9/21/2018 7:05 am
I think solutions for the entire concept described in the original post by 'thought' is probably becoming needed far more than simple outliners. We live in a world of mixed-media, and the need for instant access, far more than many of us simply need to file away writing in their proper tree structure (or whatever). Bringing together all of this content under one piece of software is probably not going to ever be a solution because of the enormity of the project. Because of this, I think we should be looking to regular file/folder structures in our preferred operating system and simply find a pleasant-to-look-at "front-end" that would serve as the Google of our local content - whatever digital content that might be.
Think about the internet and how it is organized. Right, it's not. Some things for some content certainly are (Wikipedia springs to mind), but on the whole it's just "out there." It didn't take long to discover that Yahoo's Directory of the World Wide Web was a mess and what was needed was what is still the "front-end" of the web today -- Google.com. We simply search and find (usually). But, we can bring intelligent naming, tagging, etc. to the locally stored content and bring some sense of order and near certainty that the front end search will being it up in milliseconds.
I like Schmid's Notebooks maybe for the simple reason that it relies on the regular file system and brings things in a pleasant fashion to the notebook.
The original post described a need we see more and more.
Think on this:
- All content in regular files and folders
- Intelligently named for easy searching
- Super-tagging or meta-names
- Anybody who has used the free search program "Everything' knows how fast that local content, of all kinds, can be cataloged and pulled up - as you type! ( https://www.voidtools.com ) Amazing software.
- Maybe there is a gleam of a solution that is as simple as utilizing our preferred tools to create our content, while using a lightning fast front-end (like 'Everything') to find and retrieve ALL of our content.
- Maybe changing the mindset from trying to finely tune the organization of our content in Yahoo-like directories of old, to a Google for our own stuff.
I know the above doesn't address every issue described in the OP. What it *could* do is get us thinking about a change of mindset and think about how we can work within a search system (and allow for tabbed searching to bring up multiple pieces of content) while being able to connect one piece of local mixed-media content to the other. It may sound half-baked because, well, it is. But, I leave it here as food for thought. In other words, it's an idea to a problem, it seems more and more of us are having, which is how we retrieve all of the mixed-media content we collect from simple text files, to images, HTML, video, presentations, on and on, without it all being in some proprietary enormous database. Not a "Room of our Own" but a "Google of our Own."
Think about the internet and how it is organized. Right, it's not. Some things for some content certainly are (Wikipedia springs to mind), but on the whole it's just "out there." It didn't take long to discover that Yahoo's Directory of the World Wide Web was a mess and what was needed was what is still the "front-end" of the web today -- Google.com. We simply search and find (usually). But, we can bring intelligent naming, tagging, etc. to the locally stored content and bring some sense of order and near certainty that the front end search will being it up in milliseconds.
I like Schmid's Notebooks maybe for the simple reason that it relies on the regular file system and brings things in a pleasant fashion to the notebook.
The original post described a need we see more and more.
Think on this:
- All content in regular files and folders
- Intelligently named for easy searching
- Super-tagging or meta-names
- Anybody who has used the free search program "Everything' knows how fast that local content, of all kinds, can be cataloged and pulled up - as you type! ( https://www.voidtools.com ) Amazing software.
- Maybe there is a gleam of a solution that is as simple as utilizing our preferred tools to create our content, while using a lightning fast front-end (like 'Everything') to find and retrieve ALL of our content.
- Maybe changing the mindset from trying to finely tune the organization of our content in Yahoo-like directories of old, to a Google for our own stuff.
I know the above doesn't address every issue described in the OP. What it *could* do is get us thinking about a change of mindset and think about how we can work within a search system (and allow for tabbed searching to bring up multiple pieces of content) while being able to connect one piece of local mixed-media content to the other. It may sound half-baked because, well, it is. But, I leave it here as food for thought. In other words, it's an idea to a problem, it seems more and more of us are having, which is how we retrieve all of the mixed-media content we collect from simple text files, to images, HTML, video, presentations, on and on, without it all being in some proprietary enormous database. Not a "Room of our Own" but a "Google of our Own."
MadaboutDana
9/21/2018 12:44 pm
Agreed with the personal Google thing. It's why I still use FoxTrot Pro on my Mac - vastly more powerful (and useful) than Spotlight, and allows me to build as many categories as I like. Or search on all of them, with hits clearly highlighted in documents, various ranking options, etc.
MadaboutDana
9/21/2018 12:44 pm
Agreed with the personal Google thing. It's why I still use FoxTrot Pro on my Mac - vastly more powerful (and useful) than Spotlight, and allows me to build as many categories as I like. Or search on all of them, with hits clearly highlighted in documents, various ranking options, etc.
Stephen Zeoli
9/21/2018 6:18 pm
I feel like a broken record, but I think you should look at TheBrain. No, it isn't open source, but it has a 20-year track record. It is similar to a wiki in that you can create as many links as you want among the "thoughts" in your brain. You can attach any type of file to your thoughts, so stick with RTF and plain text attachments and you're future proof. All the files are stored in their original format and are searchable from within TheBrain. Internal searches are fast.
The file attachments are stored in Brain folders, which are just like any other folders on your computer. Desktop searches will find them, so they will be there even if TheBrain suddenly disappears.
No, TheBrain isn't cheap, but if you're organizing your whole business with it, the cost is reasonable -- in my view.
Just my two cents.
Steve Z.
The file attachments are stored in Brain folders, which are just like any other folders on your computer. Desktop searches will find them, so they will be there even if TheBrain suddenly disappears.
No, TheBrain isn't cheap, but if you're organizing your whole business with it, the cost is reasonable -- in my view.
Just my two cents.
Steve Z.
thouqht
9/21/2018 7:00 pm
For anyone interested, I've decided to go with org-mode (via spacemacs) combined with github & dropbox. Big ol' learning & configuration investment, but since I have a programming background and am already a heavy vim user this makes sense.
In terms of future proof text file manipulation, I don't think anything else offers a more robust setup.
Pros:
- replaces a large number of other tools
- best in-class text input & manipulation
- locate and open anything extremely rapidly (helm projectile is amazing)
- powerful export options
- open source with large community
- super powerful version control via git/github
- completely customizable
- allows for easy interface with advanced text tools (grep, awk, sed etc.) should I ever need them
Cons:
- Very big upfront configuration & learning process (*extremely* big if you don't have a background in this side of the world)
- Wiki & crosslinking ability is less robust than dedicated wiki software or things like The Brain
- Not for teams (unless they are programmers)
- Not as smooth for handling lots of non-text attachments
And for those who are interested org-mode basically does:
- Outlining
- Time tracking
- Todo list / project management software (as long as you are working solo)
- distraction free writing
- high level writing (can act a lot like scrivener)
- Basic spreadsheets
- Journal
- Slides & presentation
- Calendar and even email (if you want it to)
From where I stand, long-term ability to massively simplify and future proof my workflow outweighs the learning and customization.
I'm slowly making the transition, moving one piece over bit by bit. Is working well and is honestly quite fun.
Shoutout to Jan S. for the recommendation as I wasn't even thinking of this because my past attempt to do this was too overwhelming.
In terms of future proof text file manipulation, I don't think anything else offers a more robust setup.
Pros:
- replaces a large number of other tools
- best in-class text input & manipulation
- locate and open anything extremely rapidly (helm projectile is amazing)
- powerful export options
- open source with large community
- super powerful version control via git/github
- completely customizable
- allows for easy interface with advanced text tools (grep, awk, sed etc.) should I ever need them
Cons:
- Very big upfront configuration & learning process (*extremely* big if you don't have a background in this side of the world)
- Wiki & crosslinking ability is less robust than dedicated wiki software or things like The Brain
- Not for teams (unless they are programmers)
- Not as smooth for handling lots of non-text attachments
And for those who are interested org-mode basically does:
- Outlining
- Time tracking
- Todo list / project management software (as long as you are working solo)
- distraction free writing
- high level writing (can act a lot like scrivener)
- Basic spreadsheets
- Journal
- Slides & presentation
- Calendar and even email (if you want it to)
From where I stand, long-term ability to massively simplify and future proof my workflow outweighs the learning and customization.
I'm slowly making the transition, moving one piece over bit by bit. Is working well and is honestly quite fun.
Shoutout to Jan S. for the recommendation as I wasn't even thinking of this because my past attempt to do this was too overwhelming.
Paul Korm
9/22/2018 11:37 am
It's nice to see a report like this when someone reaches the end of the quest and takes the time to post info about the hows and whys of their decision.
Thanks for this -- may the CRIMP be with you :-)
thought wrote:
Thanks for this -- may the CRIMP be with you :-)
thought wrote:
For anyone interested, I’ve decided to go with org-mode (via spacemacs) combined with github & dropbox. Big ol’ learning & configuration investment, but since I have a programming background and am already a heavy vim user this makes sense.
1
2
