foreign-thon/foreignthon-docs
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added architeceutre and releasing guides

Browse Source
This commit is contained in:
Keshav Anand
2026-05-21 14:39:13 +00:00
parent e763c9fbc6
commit 943791cb07
2 changed files with 294 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

159
docs/dev/architecture.md
View File

@@ -1 +1,160 @@
# Architecture # Architecture
This page describes how `foreignthon-core` works internally.
---
## Pipeline
```
source.xx.py
│
▼
_check_shebang() ← reads "# foreignthon: xx" if present
│
▼
load_pack(lang_code) ← discovers + validates the JSON pack
│
▼
_apply_postfix_syntax() ← rewrites "expr @@keyword:" lines
│
▼
_swap_tokens() ← tokenizer pass: replaces NAME tokens
│
▼
standard Python string ← ready to compile or write to disk
```
---
## Module overview
| Module | Responsibility |
|---|---|
| `transpiler.py` | The engine — postfix rewriter and tokenizer pass |
| `pack.py` | Pack discovery, loading, and validation |
| `cli.py` | Click commands (`fpy run`, `fpy compile`, etc.) |
| `errors.py` | Bilingual exception hook |
| `template.json` | Canonical set of all keywords/builtins a pack must cover |
---
## Tokenizer-based translation
ForeignThon uses Python's standard `tokenize` module rather than regex or AST manipulation.
`tokenize.generate_tokens()` splits source code into typed tokens. ForeignThon only looks at `NAME` tokens — identifiers. It replaces any `NAME` token whose string appears as a key in the active pack mapping. All other token types (strings, comments, operators, numbers) pass through unchanged.
This gives three important guarantees:
1. **Strings are safe.** A keyword inside `"..."` or `f"..."` is a `STRING` token, never a `NAME` — it is never touched.
2. **Comments are safe.** Comment tokens are passed through verbatim.
3. **Variable names are safe.** A variable like `si_condition` contains `si` only as a substring; as a `NAME` token it is `si_condition`, which is not in the mapping.
The whitespace between tokens is preserved by tracking `(row, col)` positions and copying the gaps from the original source.
---
## Pack discovery
Language packs register themselves using Python [entry points](https://packaging.python.org/en/latest/specifications/entry-points/):
```toml
# in foreignthon-es/pyproject.toml
[project.entry-points."foreignthon.langs"]
es = "foreignthon_es"
```
`pack.py` calls `importlib.metadata.entry_points(group="foreignthon.langs")` at runtime to discover all installed packs. Installing a pack is sufficient — no configuration file needs to be edited.
Each pack module must expose:
```python
def get_pack_path() -> Path:
return files(__name__) / "xx.json"
```
The core calls `get_pack_path()` to locate the JSON, loads it, and validates that all required sections are present.
Results are cached with `@lru_cache` so each pack is loaded at most once per process.
---
## Pack mapping
Four sections of the JSON are merged into a single flat dict for translation:
```python
mapping = {}
mapping.update(pack["keywords"])
mapping.update(pack["builtins"])
mapping.update(pack["exceptions"])
mapping.update(pack["stdlib"])
```
The merged mapping is `{ foreign_word: english_word }`. It is passed directly to `_swap_tokens()`.
If two sections define the same foreign key, later sections win (stdlib last). In practice this does not occur because pack authors ensure uniqueness.
---
## Postfix syntax (`@@`)
The `@@` operator is a source-level pre-processing step that runs **before** tokenization.
A line like:
```
x > 0 @@si:
escribir(x)
```
is rewritten to:
```
si x > 0:
escribir(x)
```
The rewriter uses a regex that matches `(.+?)@@(<keyword>)` and moves the keyword to the front. It only operates on lines that contain `@@`, preserving indentation and line endings.
`@@` is never valid Python and never appears in the tokenizer output.
**Decompile direction:** `fpy decompile --postfix` does the reverse — it looks for lines of the form `foreign_kw expr:` where `foreign_kw` is in the pack's `postfix_keywords` list, and rewrites them to `expr @@foreign_kw:`.
---
## Bilingual error hook
`errors.py` installs a custom `sys.excepthook` before running user code:
1. On exception, it looks up the exception type name in the pack's `exceptions` section (reverse map: English → foreign).
2. It looks up a translated message in `error_messages`.
3. It prints `[XX] ForeignName: translated_msg` then `[EN] EnglishName: original_msg`.
4. It always calls `traceback.print_exception()` afterwards so the full traceback is shown.
Tracebacks point to the original `.xx.py` file. This is achieved by populating `linecache.cache` with the original source before `exec()`-ing the compiled code, so Python's traceback machinery reads the right lines.
---
## Custom pack override
When `.foreignthon.toml` declares `custom_pack = "path/to/custom.json"`:
- If the custom JSON has `meta.code` set, it is treated as a **standalone pack** and used directly.
- If `meta.code` is absent, it is treated as an **override** — it is merged on top of the installed pack, replacing only the keys it defines.
The CLI (`cli.py`) handles this in `_load_effective_pack()` by walking up the directory tree to find `.foreignthon.toml`.
---
## File naming and language detection
Language detection order (highest priority first):
1. `--lang` CLI flag
2. Shebang comment `# foreignthon: xx` on the first line
3. Double extension `.xx.py` → `xx`
4. Fallback to `"en"` (no-op — English is Python)
`_detect_lang()` and `_check_shebang()` in `transpiler.py` implement steps 3 and 2 respectively. Step 1 is handled by the `run` command in `cli.py`.

135
docs/dev/releasing.md
View File

@@ -1 +1,136 @@
# Releasing # Releasing
This page covers how to release `foreignthon-core` and how language pack maintainers should respond to a new core release.
---
## Core release (`foreignthon`)
### 1. Update the version
Version is set in `pyproject.toml`:
```toml
[project]
name = "foreignthon"
version = "0.5.4" # bump this
```
ForeignThon follows [Semantic Versioning](https://semver.org/):
| Change | Version bump |
|---|---|
| Bug fix, no API change | Patch (`0.5.3` → `0.5.4`) |
| New feature, backward compatible | Minor (`0.5.x` → `0.6.0`) |
| Breaking change (removes a key, changes CLI contract) | Major (`0.x.y` → `1.0.0`) |
### 2. Update `template.json` if needed
If the release adds new keywords, builtins, or sections, update `template.json` to include them. This is the canonical list all language packs are expected to cover. Document what changed in your commit message and PR so pack maintainers know what to add.
### 3. Run CI locally
```bash
pytest
ruff check src/
ruff format --check src/
```
All must pass.
### 4. Tag and push
The Gitea CI publishes to PyPI on any tag matching `v*`:
```bash
git tag v0.5.4
git push origin v0.5.4
```
The `publish.yml` workflow builds the wheel and uploads it using the `PYPI_TOKEN` secret configured in Gitea.
### 5. Verify
```bash
pip install --upgrade foreignthon
fpy --version
```
---
## Language pack release (`foreignthon-xx`)
Each language pack is independently versioned and released.
### Responding to a core release
When `foreignthon-core` adds new entries to `template.json`:
1. Add the new keys to your `xx.json` with translations.
2. Run `fpy pack src/foreignthon_xx/xx.json` to validate.
3. Run `pytest tests/` to confirm all checks pass.
4. Bump your pack's version in `pyproject.toml`.
5. Tag and push — the `publish.yml` workflow handles PyPI upload.
Update the `foreignthon` dependency lower bound in `pyproject.toml` if your pack requires the new core version:
```toml
dependencies = ["foreignthon>=0.5.4"]
```
### Releasing independently
You can release a pack at any time — to fix a translation, add missing entries, or extend coverage. The process is the same:
```bash
# bump version in pyproject.toml
git add pyproject.toml src/foreignthon_xx/xx.json
git commit -m "v0.3.3: fix translation for 'return'"
git tag v0.3.3
git push origin v0.3.3
```
---
## Docs release (`foreignthon-docs`)
The docs site rebuilds automatically when a language pack fires its `trigger-docs.yml` workflow after a successful publish. No manual step is needed for routine pack releases.
To manually trigger a docs rebuild (e.g. after editing `foreignthon-docs` directly):
```bash
git push origin main
```
The `deploy.yml` workflow builds the MkDocs site and deploys it to [foreignthon.keshavanand.net](https://foreignthon.keshavanand.net).
---
## Gitea secrets
| Secret | Used by |
|---|---|
| `PYPI_TOKEN` | `publish.yml` in core and all packs |
| `DOCS_WEBHOOK` | `trigger-docs.yml` in language packs |
Secrets are set per-repository in **Gitea → Settings → Actions → Secrets**.
---
## Checklist
### Core
- [ ] Version bumped in `pyproject.toml`
- [ ] `template.json` updated if new keys added
- [ ] `pytest` passes
- [ ] `ruff check` passes
- [ ] Tagged `v*` and pushed
- [ ] PyPI upload confirmed
### Language pack
- [ ] New `template.json` entries translated in `xx.json`
- [ ] `fpy pack xx.json` passes
- [ ] `pytest tests/` passes
- [ ] Version bumped
- [ ] `foreignthon` dependency lower bound updated if required
- [ ] Tagged and pushed