You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
82 lines
3.5 KiB
Markdown
82 lines
3.5 KiB
Markdown
# Crawl lists of active Session Communities
|
|
|
|
## What does this site do?
|
|
|
|
This script crawls known sources of published Session Communities,
|
|
queries their servers for available information and
|
|
displays this information as a static HTML page.
|
|
The results of this can be viewed on <https://sessioncommunities.online/>.
|
|
|
|
## What is Session?
|
|
|
|
Session is a private messaging app that protects your meta-data,
|
|
encrypts your communications, and makes sure your messaging activities
|
|
leave no digital trail behind.
|
|
<https://getsession.org/>
|
|
|
|
## Details
|
|
|
|
### Which sources are crawled?
|
|
|
|
Currently this script crawls the following sites:
|
|
|
|
- <https://github.com/GNU-Linux-libre/Awesome-Session-Group-List>
|
|
- <https://lokilocker.com/Mods/Session-Groups/wiki/Session-Open-Groups>
|
|
- <https://session.directory/>
|
|
|
|
Additionally, [a few other servers are hardcoded](php/servers/known-servers.php).
|
|
|
|
### How does this work?
|
|
|
|
The [`update-listing.php`](php/update-listing.php) script invokes the following two PHP scripts: [`fetch-servers.php`](php/fetch-servers.php) to query available servers, and [`generate-html.php`](php/generate-html.php>) to generate the static HTML.
|
|
|
|
The querying logic consists of these steps:
|
|
|
|
1. Fetching source HTML: `query_known_sources()`
|
|
1. Extracting Session join URLs from the HTML: `parse_join_links()`
|
|
1. Building server instances from join URLs: `CommunityServer::from_join_urls()`
|
|
1. Adding known servers to list: `CommunityServer::from_known_hosts()`
|
|
1. Merging servers based on URL: `CommunityServer::dedupe_by_url()`
|
|
1. Making sure servers are online and querying rooms & pubkeys:
|
|
`CommunityServer::poll_reachable()`
|
|
1. Merging servers based on public keys: `CommunityServer::dedupe_by_pubkey()`
|
|
|
|
Static HTML is generated from the [`sites`](sites) directory to the [`output`](output) directory, which additionally contains static assets. All contents of `sites` are invoked to produce a HTML page unless they are prefixed with a `+` sign.
|
|
|
|
### Work around bad routing to Chinese servers
|
|
|
|
Depending on your location, it is possible for you to get really bad routing to
|
|
SOGS servers behind the [GFW](https://en.wikipedia.org/wiki/Great_Firewall). In this case,
|
|
the initial connection is still successful, but you'll never receive
|
|
any actual content and the retrieval attempt will simply time out.
|
|
This happens randomly. To make sure this won't affect the results, we simply
|
|
check whether the server is online (the initial connection being successful),
|
|
and then retry a lot of times with a short timeout
|
|
until we eventually get the content.
|
|
The details can be seen in [`curl_get_contents()`](php/utils/utils.php).
|
|
|
|
### Official repositories
|
|
|
|
- Codeberg: <https://codeberg.org/gravel/sessioncommunities.online>
|
|
- Lokinet Gitea (mirror): <https://lokilocker.com/gravel/sessioncommunities.online>
|
|
- GitHub (former official repository): <https://github.com/mdPlusPlus/sessioncommunities.online>
|
|
- Lokinet Gitea (former mirror): <https://lokilocker.com/SomeGuy/sessioncommunities.online>
|
|
|
|
If your favourite Session community is missing a language flag, read [this](languages/README.md) on how to add a flag.
|
|
|
|
Alternatively, support this issue on Oxen Github to make language tags a native feature:
|
|
|
|
- <https://github.com/oxen-io/session-pysogs/issues/186>
|
|
|
|
### Contributing
|
|
|
|
See [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
|
|
## Contact
|
|
|
|
To report issues, visit the Web Development Community on [caliban.org](http://sog.caliban.org/r/webdev).
|
|
|
|
Contact [gravel](https://codeberg.org/gravel/gravel) on Session as `gravel` (Registered Session ID).
|
|
|
|
Contact SomeGuy on Session as `someguy` (Registered Session ID).
|