KSF Progress Report

Since last December, the Knowledge Standards Foundation has made great progress, so we have a lot of big news, from the top of this newsletter to the bottom!

1. Explainer video premieres tomorrow @ 11 a.m. EST:
The Universal Network of Encyclopedias: A Progress Report

Premieres Tuesday, 11 a.m. Eastern; about 50 minutes. YouTube or Rumble. Add it to your calendar! Think of it as a well-produced video white paper. For months, we have been busy developing this in-depth explanation of the network we are building. The network has many moving parts, so there is a lot to explain; but it should be time well spent, particularly for developers and reference publishers. The network we are building is going to be, in my honest opinion, a model for future decentralized open content networks. This video is probably the best way to learn about it. I and possibly other KSF folks will be on hand during the premiere on YouTube to chat. Yes, we don’t like YouTube either, but…you know. I will check in on the Rumble premiere too.


2. About $30K raised since last November; please help!

We raised over $10K from 111 different people since November, with $18,000 in 2x matching funds from FUTO (for the donations through December); we are awaiting a much bigger donation from FUTO. We want to thank everyone who donated. Suffice it to say that, considering how many underpaid staff and volunteers have been working hard on the encyclosphere and Minifeed projects, mostly in their spare time, it’s amazing what we have achieved. You can see this both in the video linked above and in the updates given below.

While we expect a significant donation from FUTO in the next few weeks, we cannot assume they will continue their support next year. We will be seeking out other large donors. Smaller donors: please help, if you want to see the encyclosphere develop into a mature system. Our vision is to make all the free encyclopedic articles online available in the same, useful format; with metadata about proprietary articles searchable as well; distributed in a decentralized way; on which any reasonably skilled developer can easily build fast specialized or general encyclopedia search engines and readers, based on our free software—see below. The result will be significantly more useful to the world than Wikipedia alone.

3. ZWIMaker created, and tested on Citizendium

ZWIMaker is an extension of MediaWiki, the software that runs Wikipedia and many, many smaller encyclopedias. This software allows wiki administrators to automatically convert articles to ZWI files (i.e., the KSF’s standard encyclopedia file format), download them if they prefer, and upload them to the EncycloReader aggregator. A later planned version of the extension will enable site administrators to export all articles, or all that meet certain conditions, in this way. There will also be an option to export articles automatically whenever they’re updated. With this software, we are beginning to make it possible for encyclopedias (at least, those that run MediaWiki) to contribute their articles to the encyclosphere.

In order to test this extension “in the wild,” our lead developer, Dr. Sergei Chekanov, as a volunteer, thoroughly updated the software running Citizendium.org. Editor-in-Chief Pat Palmer worked with Sergei to test some of the initial issues with ZWIMaker.

Sergei has also created a similar plugin for DokuWiki, another leading wiki software system; the plugin is called ZWIdoku. This has been tested on HandWiki, a STEM encyclopedia, mostly curated and edited versions of Wikipedia articles.

4. EncycloReader bigger than ever

Articles added via ZWIMaker from Citizendium, and via ZWIdoku from HandWiki, can now be read in EncycloReader, which itself continues to be maintained and improved. The encyclosphere’s first aggregator (of two) now features over 100,000 articles. (You can view the count at the top of encyclosphere.org, and you can view our log of articles created for the encyclosphere on this page; from the same page, you can download the ZWI file for any article in the database.) We would and could already have millions of articles from hundreds of sources, but we are still working out the kinks in the network software and protocols before we take the plunge into thoroughgoing crawling of all encyclopedic content online. We would also prefer publishers to push their own content to the encyclosphere rather than aggregators crawling it, but this is not strictly speaking necessary.

5. EncycloSearch updated, and transferred to new server

The KSF is now renting a new server, which is being used to host EncycloSearch. At the same time, we have replaced the old version, which was just a meta-search engine, with a version that searches its own copies of encyclopedia articles. So EncycloSearch is much faster. This has required that developer Henry Sanger create a second encyclosphere aggregator. It is a part of our methodology for making a bona fide decentralized network that two developers independently create two different search engines, readers, and aggregators, so that they can explore how best to standardize and exchange data. We are even now exploring the best methods for formatting encyclosphere databases.

We are also using the new server to host an organizational instance of NextCloud, the NAS software that provides (among much else) free, high-quality groupware for filesharing, internal chat, email, calendar, notes, project management, organizational passwords, etc. As an organization, we urge developers particularly, and “advanced” computer users in general, to install their own NAS systems, buy their own domain names, and cast off the tender mercies of Big Tech.

6. EncycloTorrent: a new encyclosphere aggregator, REST API, and BitTorrent seeder

EncycloSearch now runs on top of a new encyclosphere aggregator, built from scratch: EncycloTorrent. This is an ambitious new project to make the content of the encyclosphere available via the P2P BitTorrent network, from our faster new server. While it is probably not feasible to seed each article individually, we use the BitTorrent network’s webseeds functionality to make free articles (and metadata about proprietary articles) accessible via BitTorrent. “Under the hood,” the software spins up an HTTP server that acts as an aggregator. The aggregator manages a database of ZWI files; you can use either server commands or a REST API to add files to the database from various sources of your choice (command line only, but we will fix this soonish). Once this is set up, you can download files from the database using the REST API and/or BitTorrent. Theoretically, developers could create brand new encyclopedia apps with the encyclosphere’s data via this API, without hosting any articles themselves; we are interested in working with developers who want to try this.

7. Minifeed v2 ready for beta-testing

Minifeed is our social media project aiming to let people use WordPress to create their very own social media feed. Version 2 is ready to onboard beta testers; email me (just hit “reply” to this email) if interested. It’s totally redone from v1. The big new feature is that it allows you to follow other Minifeed installs (any RSS feed, actually) and tracks things like the numbers of accounts following and being followed by you. It’s not quite ready yet for a public demo, but—soon!

8. Assorted notes

  • I was on Tucker Carlson Today a few weeks ago, talking to the pundit for an hour about Wikipedia, Internet freedom, and the encyclosphere. We have a copy, but don’t share it (or Fox will have it deleted). The show was about 2/3 about Wikipedia and 1/3 about the Internet and KSF projects.
  • One of our board members, Christian Gribneau, is behind a new W3C DID implementation called PSQR. This will enable us to create and sign self-owned, self-hosted identities, and use them to sign ZWI files (catch the video for details) as well as social media posts. Among other things, his company, Ology, has created a WordPress plugin that we intend to use to sign Minifeed posts. We will also be implementing support for DID:PSQR at several levels throughout the network.
  • A new regular attendee to our Friday meetings has kindly volunteered to author another WordPress plugin. It will enable those who use WordPress to manage encyclopedias—there are quite a few—to create ZWI files out of their articles, sign them, and push them to the encyclosphere. We hope to work with a few members (mostly historians) of the Digital Encyclopedia Group; this might turn out to be the easiest way for many bloggers to add their contributions to the encyclosphere.
  • We have restarted conversations with Everipedia, so that they might set up the first truly independent encyclosphere aggregator, and perhaps put encyclosphere data on the blockchain; and MyMentor, who might well become the first specialist reference site to use EncycloReader or EncycloSearch. We will, of course, reach out to many other reference publishers when everything is quite ready for them.

9. Plans

We have big plans for the coming year. Without going into detail, in the coming year we intend to:

  • Improve the design, search relevance, and data quality of both EncycloReader and EncycloSearch, and add a reader to EncycloSearch. As we have been so busy with backend/networking issues, we have neglected features that the public care about most. That will change soon!
  • Settle on a fairly final version of the ZWI standard, including the forthcoming “Attic” feature which will standardize article revision history and support for DID signing by both authors and publishers.
  • Incorporate all of the articles from Wikipedia, Britannica, and many other encyclopedias, large and small, free and proprietary.
  • Run aggregators, and make it easy for others to start aggregators, that include all of that data, and on which small but fast apps can be developed.
  • Support a variety of encyclopedias that regularly publish and update signed copies of their data (or, if proprietary, just metadata).
  • Develop Minifeed so that totally self-owned WordPress miniblogs become interoperable with the Fediverse and Twitter.
  • And much more.

The above represents by no means the total extent of our work and our plans, but just the highlights.

Again, if you want the encyclosphere to bloom faster, please donate. If you are a developer inspired by our work and are interested in developing any aspect of this project, join us on our Slack group (that’s an invite link; and yes, we will eventually move from Slack to FOSS; Slack is just a concession to familiarity and ease of use, as we get started).

All the best,
Larry Sanger

By Larry Sanger

See this page for my bio. Welcome to this site! Thanks for being here!

Leave a comment