sessioncommunities.online/README.md

# Crawl lists of active Session Communities

## What does this site do?

This script crawls known sources of published Session Communities,
queries their servers for available information and
displays this information as a static HTML page.
The results of this can be viewed on <https://sessioncommunities.online/>.

## What is Session?

Session is a private messaging app that protects your meta-data,
encrypts your communications, and makes sure your messaging activities
leave no digital trail behind.
<https://getsession.org/>

## Details

### Which sources are crawled?

Currently this script crawls the following sites:

- <https://github.com/GNU-Linux-libre/Awesome-Session-Group-List>
- <https://lokilocker.com/Mods/Session-Groups/wiki/Session-Open-Groups>
- <https://session.directory/>

Additionally, [a few other servers are hardcoded](php/servers/known-servers.php).

### How does this work?

The [`update-listing.php`](php/update-listing.php) script invokes the following two PHP scripts: [`fetch-servers.php`](php/fetch-servers.php) to query available servers, and [`generate-html.php`](php/generate-html.php>) to generate the static HTML.

The querying logic consists of these steps:

1. Fetching source HTML: `query_known_sources()`
1. Extracting Session join URLs from the HTML: `parse_join_links()`
1. Building server instances from join URLs: `CommunityServer::from_join_urls()`
1. Adding known servers to list: `CommunityServer::from_known_hosts()`
1. Merging servers based on URL: `CommunityServer::dedupe_by_url()`
1. Making sure servers are online and querying rooms & pubkeys:
`CommunityServer::poll_reachable()`
1. Merging servers based on public keys: `CommunityServer::dedupe_by_pubkey()`

Static HTML is generated from the [`sites`](sites) directory to the [`output`](output) directory, which additionally contains static assets. All contents of `sites` are invoked to produce a HTML page unless they are prefixed with a `+` sign.

### Work around bad routing to Chinese servers

Depending on your location, it is possible for you to get really bad routing to
SOGS servers behind the [GFW](https://en.wikipedia.org/wiki/Great_Firewall). In this case,
the initial connection is still successful, but you'll never receive
any actual content and the retrieval attempt will simply time out.
This happens randomly. To make sure this won't affect the results, we simply
check whether the server is online (the initial connection being successful),
and then retry a lot of times with a short timeout
until we eventually get the content.
The details can be seen in [`curl_get_contents()`](php/utils/utils.php).

### Official repositories

- Codeberg: <https://codeberg.org/gravel/sessioncommunities.online>
- Lokinet Gitea (mirror): <https://lokilocker.com/gravel/sessioncommunities.online>
- GitHub (former official repository): <https://github.com/mdPlusPlus/sessioncommunities.online>
- Lokinet Gitea (former mirror): <https://lokilocker.com/SomeGuy/sessioncommunities.online>

If your favourite Session community is missing a language flag, read [this](languages/README.md) on how to add a flag.

Alternatively, support this issue on Oxen Github to make language tags a native feature:

- <https://github.com/oxen-io/session-pysogs/issues/186>

### Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).

## Contact

To report issues, visit the Web Development Community on [caliban.org](http://sog.caliban.org/r/webdev).

In case the issue cannot be resolved publicly, contact [gravel](https://codeberg.org/gravel/gravel) on Session as `gravel` (Registered Session ID).