Q4 2023 Newsletter

Dear KSF Community,

The network of all the encyclopedias is taking shape, and we continue to make steady progress. Across the affiliated Encyclosphere databases, we now have close to 3 million ZWI files from 66 encyclopedias. Our biggest news from the last few months is that it is easier than ever for developers to get involved, especially thanks to ZWINode.

1. ZWINode is taking off.

ZWINode (code heredemo here) is now in version 1.3, which means it is increasingly ready for prime time. We thank Dr. Sergei Chekanov for all his work on this. It allows organizations to host their own encyclopedia search engine/readers, with an increasingly sophisticated article publishing system. (This is a feature our other OSS Encyclosphere node software, EncycloEngine, does not yet have.) The publishing system’s article editor is designed to be extremely easy to use, with most expected features and even MathJax (i.e., LaTeX support). It is easy to import collections (i.e., encyclopedias) from other Encyclosphere aggregators, as well to export to the other aggregators. Your install becomes another aggregator itself, another node on the Encyclosphere!

It is fair to say that ZWINode is another approach to encyclopedia publishing, but one that is fully networked. We hope you will consider using it.

2. We’re inviting new developers!

Developers, Larry has a new blog post, “Return of the Coders“, which explains why and how you can get involved with the Encyclosphere.

In summary: Devs know better than anyone that the reason Big Tech systematically violates our free speech, privacy, and autonomy is that they run giant centralized platforms. Devs also know that the solution is to develop open, decentralized, standards-driven alternatives for each vertical. But an increasingly-evident problem is the proliferation of competing standards. (This was always evident, it’s just more so now, when it actually matters.) Arguably, the only vertical of which this isn’t true is encyclopedias, in which there is one open network, the Encyclosphere, and one file standard, the ZWI file format. The post, tailored for developers, goes into why this is so important and gives some technical details on how to get started. For one thing, we really need more people generating new ZWI files, but there are so many possible projects. Want to get involved? Join us on our Mattermost and on our weekly video conference, Fridays at 2 p.m. Eastern.

3. We added 30 new geographical encyclopedias.

We’ve connected with the Digital Encyclopedia Group (DEG), an association of professional historians (mostly) from around America who manage regional resources like Handbook of Texas and Encyclopedia of Greater Philadelphia. We’ve added 30 of their high-quality encyclopedias to the Encyclosphere. Here’s how to check them out:

  • Go to EncycloSearch.org.
  • Click “Categories”, right under the search box.
  • Press the red “Disable all” button.
  • Then check “Geography & Politics”. (Note, this includes Ballotpedia, also professionally edited, but not part of the DEG. You can remove it via “Categories” > the “View all sources” link, at the bottom.)
  • Now you can search 30 DEG encyclopedias! Neat searches to try: “Springfield”, “Houston”, and “peaches”.

Tim would love your feedback about what we’ve done with the DEG encyclopedias. Call or txt him in his KSF office: +1.614.908.0729. Do you know of other digital encyclopedias anywhere in the world? Please see the section about the Encyclopedia Listing below.

4. We created a new organizational wiki.

Some people complained to us that it was hard to grok everything the Knowledge Standards Foundation was up to. Others complained that we had no easy way to manage, or at least track, all that we are doing.

So we decided to create an organizational wiki! Here:

wiki.encyclosphere.org

It’s not an encyclopedia. It’s just more detailed documentation of what we are up to, aimed at participants (and potential participants), rather than at casual visitors (who we’ll continue to send to encyclosphere.org). It’s brand new, so it’s not finished. But it’s already pretty useful and has lots of links. Maybe it’ll help get you involved?

5. We set up two new aggregators.

Thanks and congratulations are due to two new Encyclosphere aggregators:

Being aggregators, they collect ZWI files from other sources. These two (along with ER & ES) are not the only things we might call “Encyclosphere nodes,” however. For such a list, we should also include Louis’ Wikipedia Deathrow and Oldpedia. But the latter two projects only produce/publish new ZWI files; they don’t aggregate files from elsewhere.

6. We added two new backup systems.

We added two new backup servers after the host for several of our websites had a sudden, severe, multi-week outage. One is public:

  • Sergei’s new “EncycloBackup” has the most complete collection of ZWI files from all sources, here.
  • Henry built a 22TB server from components, and it now sits behind Larry’s desk. As I send this out, it is busily slurping up all KSF data, both ZWI files as well as backups of the KSF’s Mattermost, NextCloud, etc.

Due to existing redundancies, there was never any danger of losing files, but these multiply-redundant backups will put our minds at ease.

7. We made various smaller improvements.

We have continued to make progress on existing projects:

  • EncycloEngine is getting a rewrite from Java to JavaScript (TypeScript), providing a JS library for client-side ZWI file manipulation. (This is a months-long project.)
  • Our WordPress plugin, EncycloShare, allows you to push articles from your blog to the Encyclosphere. (Do it!) It now features thumbnails (example: look for the “SP” logo near the top) for encyclopedia collections. This entailed a slight update to the ZWI standard, which should now include small copies of encyclopedia logos in the ZWI files.
  • We’ve resolved issues with pushing articles from Shelly’s aggregator to EncycloReader.
  • We have demonstrated that it should be possible to take a plaintext OCR output file of an old public domain encyclopedia and, using nothing but text processing (not AI), identify all the titles of the articles. This is the sort of work we could really use developer help with, though. This one feature, though tricky to implement perfectly, would open the floodgates, potentially enabling Oldpedia to generate thousands and thousands of usable (but probably still imperfect and in need of editing) newly-digitized encyclopedia articles from public domain sources. One of our projects is to grade encyclopedia volumes found on Archive.org. We have graded 623 so far, and it turns out that, so far, 124 of these have a grade of A or B, which means they’ll be good for modern, AI-driven OCR.

8. Reminders.

Your continued support and contributions are invaluable to our shared mission to advance knowledge standards that will help the encyclosphere realize its full potential.

Thank you for being an integral part of the Knowledge Standards Foundation.

A few reminders:

  1. Please donateWe need your support. We do not have anything like an endowment and our current account is dwindling.
  2. We have merch! If you want to show your support for the project (KSF gets a cut of the proceeds), please pick up an Encyclosphere mug, t-shirt, hoodie, or cap at our online store.
  3. Help expand our Encyclopedia Listing: librarian and database help requested. The KSF has compiled a database of encyclopedias (779 entries vs. 743 last quarter). We need your help adding entries and improving the web app front-end so that we can release it to the public. The main purpose of this database is to accelerate populating the Encyclosphere. We are using the database to give our developers ideas for what to add to Encyclone, EncyloReader.org, and EncycloSearch.org. Furthermore, the Old Encyclopedia Digitization Project (OEDP) digitizes public domain paper encyclopedias so that they, too, can be added to Encyclone (via Oldpedia.org) and, from there, into the Encyclosphere. We know there are thousands of encyclopedias digitized to one degree or another in the archive.org and Google Books collections. Please respond to this email if you’re interested in helping with the database. Or call or txt Tim at +1.614.908.0729.
  4. Previous donors! The donor “choose articles to support” server is down due to host outage, almost immediately after we announced it. The server is back up, so you should be able to use it. This experimental project allows KSF donors to provide what names they would like to include in public domain ZWI files, one name per file per $1 donated. If people use the system more, we’ll support it more.

Finally, please check out our active projects page for further opportunities to volunteer (including development!), then join us on our Mattermost! And we want to acknowledge and thank all of you who are also grassroots financial supporters. We are grateful for your one-time and monthly contributions! If you’ve been thinking about making a donation, now is a good time to do so!

We meet every Friday at 2pm U.S. Eastern time: encyclosphere.org/meetmod. We hope you’ll join us!

Warm regards,

Encyclosphere Team

By Larry Sanger

See this page for my bio. Welcome to this site! Thanks for being here!

Leave a comment